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ABSTRACT 

Decision Oriented Evaluation System (DOES) for 
community development training presents a system for training 
evaluation in prototypic form. This handbook provides a comprehensive 
overview of traini.ng evaluation methodology as well as details on 
specific functionrs involved in the training evaluation process. This 
model for evaluation is broken into three major categories, i.e., 
three modes of evaluation: (1) training, (2) job performance, and (3) 
community impact. Each of these three modes of evaluation are 
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outcome evaluation. (Author/BW) 
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SECTION I 

INTRODUCTION The administrator of community-related programs plays 

a unique role. He functions as a vital decision-making 
link between two complex systems, the political/pro- 
fessional world of funding agencies and the community 
itself. In that position the administrator needs an 
evaluation methodology which helps him make intelligent 
decisions related to both of these worlds. 
Puirpose The ongoing process of community development training 

necessitates the operation of training systems which are 
congruent with the needs of communities. If training 
is to be relevant to the long-range developmental objec- 
tives of community administrators and public officials, 
they need evaluation systems which: (1) provide fast 
and effective decision-making information; and (2) in- 
sure that this information is appropriate to the decisions 
that have to be made. 
DOES Such an evaluation system, called a Decision Oriented 

Evaluation System, is presently under development by 
Project CATE (Community Affairs Training Evaluation) 
at the University of Texas Research and Development 
Center for Teacher Education. This handbook, which 
presents the DOES system in prototypic form, is designed 
to be expanded and refined as thp system undergoes 
further development and testing* 
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The Decision Oriented Evaluation System treats the 
processed of training and evaluation as complimentary, 
integrated functions; an effective training system 
will contain evaluation mechanisms as an integral part 
of its operation. Therefore, an effective evaluation 
system for training must have input into the develop- 
ment and inq)le'ientation of training activities, and 
must concern itself with the developmental objectives 
of the training system. The community or public ad- 
ministrator, then, needa access to information on both 
training and evaluation, whether he performs his own 
training and evaluation, or has them subcontracted. 
The Handbook The DOES Handbook is designed for practical and fre- 

quent use. For use in over-all training system 
development, it provides a comprehensive overview of 
training evaluation methodology; for details on spe- 
cific functions j^nvolved in the training evaluation 
process, it gives quick reference keys on pg. 9. 
Technical Report For theoretical princi;>les, rationale, recommendations, 

position papers and detaj.led documentation on design 
and development of the DOES, see Project Cate Final 
Report, 
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SECTION II 

COMPONENTS If your job is to make decisions related to training and 

OF DOES 

evaluat: m of training, your first task is to orient 
yourself to the system which gives you your information. 
There are three major categories of information needed 
for systematic evaluation. These are called modee of 
evaluation. These modes, or information bases, are 
concerned with: 
Modes (1) training 

(2) job perfoT.Tnax5ce 

(3) community impact 

training Mode 1 is concerned with determining the prerequisites for 

training evaluation, implementing procedures for observ- 
ing training, and determining skill' acquisition by trainees. 
For training evaluation to be effective, epeoific goals 
of training must be established, epeoific objectives 
must be derived from those goals, and epeoific observable 
behaviors must be specified as a function or training. 
Unleee these conditione are mat it will be exceedingly 
difficult to judge the effect of training and to conduct 
training evaluation. 

job per- Mode 2 assesses the transfer effects of training to on- 

formance 

the- job performance of trainees within a natural job 
context. Job performance evaluation is a natural outgrowth 
of training evaluation. In this regard, decision-makers 
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ask the important question, "Did training actually change 
the on-the-job performance of individuals trained?" Job 
performance evaluation is difficult unless the training 
and the job include specific observ^-able operations; even 
then, the judgment of quality becomes a rather difficult 
problem, 

community Mode 3 traces or observes the effects of training on 

impact 

conimunity impact indicators, (i.e., it attempts to answer 
the question, "Are there any observable community benefits 
which can be related to as having improved the job per- 
formance of individuals?" "Has training improved the job 
performance of individuals in a way which can be related 
to observable community benefits?"). 

Community impact evaluation is a newer and less refined 
process than either training evaluation or job performance 
evaluation. Additionally, an enormous complex of variables 
are dynamically interacting within a community, making 
attempts to isolate the effects of specific training 
difficult or impossible. For this reason, community 
impact evaluation for the most part can only implement 
indirect rather than direct measures, i.e., evaluators 
should look for changes or trends in information elicited 
as part of the "natural process records" within a 
community. 
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Stages Each of the three modes discussed above is operationalized 

in three etagee. The stages are input evaluation^ prooeaa 
evaluation and outcome evaluation, 

ioput Input evaluation refers to the gathering of data and in- 

formation which are prerequisite to the execution of both 
process and outcome evaluation. The input stage of 
evaluation operates acrods all three modes; for example, 
there is an input stage for training evaluation, one for 
job performance evaluation, and one for coramur.ity impact 
evaluation. 

process The process stage is that set of procedures which occurs 

while training evaluation, job performance measurement and 
community impact observation activities are actually taking 
place. The major objectives of the process stage of 
evaluation are: (1) to determine the observable consis- 
tency of training activities, job perfonnance observation 
and monitoring of community data with the stated objectives 
and goals; (2) to gather and feed back information to 
participants, developers and sponsors regarding the effec- 
tiveness of training, job perfonnance and community ob- 
servation; and (3) to verify Lhe training acti-/! ties , 
job performance measurements and community impact indi- 
cators are congruent with the needs, objectives and 
outcoines specified in the input evaluation stage. 
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The outcome evaluation stage Is concerned v/ith gathering 
data relative to effectiveness, that is, what returns are 
being realized for the investment in training: (1) in 
and of itself to provide a quality training process; 
(2) in terms of improving job performance; and (3) with 
respect to producing observable changes in the community. 
In summary, the input stage provides the background for 
proce^.ding in the three modes of evaluation; the process 
stage specifies the kinds of data to be gathered while 
training is in operation, when job performance is being 
observed, or when community indicators are being moni- 
tored; the outcome stage gathers data related to payoff. 
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Figure 1. DOES model, modes of evaluation 
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MODES 



TRAINING 



JOB PERFORi>iA:^CE 



COMMUNITY IMPACT 



INPUT 



pps. 11-22 



pps, 32-38 




PROCESS 



pps. 23-26 



pps. 39-40 



pps. 50-51 



OUTCOME 



pps. 27-30 



pps. 41-44 



pps. 52-56 



Figure 2. Operationalizing DOES 
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SECTION III, OPERATIONALIZING 
THE DOES SYSTEM 
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Input stage 



Part A: Training Evaluation 

Activities 
I* Needs Assessment 

2. Specification of Training Behavioral Outcomes 

3. Contextual Assessment 

4. Training Materials Selection 

5. Evaluation Design and Instrumentation 



process stage 



6. Consistency Evaluation 

7. Behavioral Observation and Feedback 



outcome stage 



5. Training Level Outcome - Performance 

9. Training Level Outcome - Knou)le./ 

10. Training Level Outcome - Awareness 

11. Training Outcomes - Affective 
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I. Neede Aeaesamenv 

A needs assessment is a discrepancy analysis - a statement of the dif- 
ference between where you new are and where you would Zifee to he. The pur- 
pose of a needs assessment in DOES is to establish a list of priorities in 
govemrnent, at all levels, that can be addressed by training. 

The following steps will assist in conf>leting a needs assessment. 

(1) Develop a plan to conduct a needs assessment; involve 
personnel from the community and as many levels of govern- 
ment as is feasible (e.g., local, county, state, federal). 

(2) Identify and list problem syn?)toms related to community 
life. 

(3) From the list developed in No. 2, select a focua for the 
needs assessment. (Note: It is assumed that the list 
of needs that could be £issessed will be far greater than 
would be possible to meet.) 

(4) Identif> possible needs assessment tools and procedures, 
(e.g., questionnaires, records, interviews) select the 
bast one(s), and obtain the participation of the community 
leaders and members in planning for gathering of infor- 
mation. 

(5) Describe the current state of affairs with respect to the 
target area of needs assessment; include job descriptions 
and current levels of proficiency in job performance. 
Describe the context within which a job is being performed 
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and the general characteristics of the workers In unat job. 

(6) Be sure that you have stated the current level of job per- 
formance in measurable, behavioral terms (e.g., it now takes 
an average of fifteen minutes for local police to receive 
and respond to .'li; emergency call). 

(7) Specify the desired conditions of job performance in mea- 
surable, behavioral terms (e.g., the average amount of time 
required for local police to receive and respond to an emer-- 
gency call shall be reduced from fifteen minutes to five 
minutes) . 

(8) Having completed discrepancy identification, identify those 
needs for which training appears to be the answer. (Note: 
Training may not alwaj's be the best solution to a problem. 
Changes in personnel selection procedures, shifts in job 
assig^iu^nt, new equipment purchase may be more viable solu- 
tions to the job perfoirmance discrepancies identified. 

(9) Establish priorities among the discrepancies identified 
as the first step in decidi train. 

(10) Decide on a course of actio* conduct training. 

(11) Provide for continual revision of the needs assessment on 
at least an annual basis. A needs assessment is never com- 
plet<^! It must continually be updated to reflect emerging 
*:cnditions in the community. 
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2. Speaifiaation of Traimng Behavioral Outcome 

The quality of training and training evaluation is largely a function of 
the scope and quality of the objectives specified. Good objectives not only 
make the job of constructing test items/situations easy, but they also help 
in the development of training materials and carrying out training programs. 
The following steps will assist in specification: 

(1) Tiie statement of training outcomes should be in the form 
of behavior that a trainee should demonstrate at the end 
of the training courae. 

(2) These terminal behavioral outcomes should be directly re- 
lated to the discrepancies Identified in the needs assess- 
ment (i.e., terminal training behaviors should represent 
the ideal state of affairs). 

(3) Training behavioral outcomes should have the following 
characteristics: 

(a) The conditions under which the beha^/ior is to be 
demonstrated must be stated (e.g., when provided 

a malfunctioning electrical circuit caused by.,,). 

(b) The specific behavior required of tne trainee must 
be stated in unambiguous action-oriented terms 
(e.g., the trainee will locate the source of the 
wiring problem and correctly affix the wiring...). 

(c) The acceptable standards must be included in the 
?*-ateinent of the objective (e.g., within a five- 
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minute period) , 

The three criteria for an adequate training objec- 
tive are summarized as follows: 

1. Conditions under which the behavior is to 
be observed must be stated* 

2. Performance required of the trainee must 
be stated in precise language. 

3. Standards for acceptable perfo-^aance must 
be listed. 

When the outcomes of training are concerned v;xth 

knowledge, or attitudinal outcomeB, the f:ollow:a».g 

sources should be consulted: 

Bloom, B. S., et al. Taxonomy of educational 
objectives: cognitive domain. New York: 
David McKay Co., Inc., 1956. 

Krathwohl, D. R. , et al. Taxonomy of educa- 
tional objectives: affective domain. New 
York: David McKay Co , Inc., 1964. 

Two general sources that provide good examples and 

guidelines for writing behavioral cutcocies of 

training are : 

Gronlund, N. E, Stating behavioral objectives 
for classroom instruction. New York: The 
Macmillan Co.^, 1970. 

Mager, R. F. Preparing instructional objectives. 
San Francisco: Fearon Publishers, 1962 5. 
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2. Contextual Aasessrmnt 

Ultimate effectiveness or ineffectiveness of a training program can 
often be dictated by circumstances or variables in the training environ- 
ment that are beyond the control of a sponsoring agency or instructors. 
It is inQ)ortant that an evaluator/sponsor anticipate in advance those 
variables by doing the following: 

(1) Provide a description of the location of the training and 
list those aspects of the environment that may assist or 
detract from training effectiveness. (Note: Many a train- 
ing program has been rendered ineffective by poor in- 
structional spaceo) 

(2) Analyze the training program and materials to be used and 
record the type of space and equipment required to conduct 
training. Check these requirements against the training 
space available. Is there enough room? 

(3) Prepare a full written description of the training popula- 
tion including age, sex, education, work experience and 
the like. If it is not possible to do this prior to 
training, then prepare an estimated description of the 
population. (Note: Training programs may fail if they 
are geared above or below the experience, educational 
level or sophistication of the trainees.) 

(4) Obtain and record background information on the instruc- 
tors who will conduct the course. 

(5) Analyze the training program and materials, including 



the instructor's guide, to prepare a short list of be- 
haviors you would expect to observe for both trainees and 
instructors. (Note: Oftentimes training programs fail 

when instructors fail to use materials or use them in a 

manner contrary to their intended use.) 

After conducting the above analyses, make a list of other 
things that you think might or could go wrong and thus 
impair the quality of the training (e.g., "The training 
may be too short in duration". "I think the trainees 
don't have enough practice time to be able to reach the 
criterion.") 

Keep an accurate record of the above to be used as a 
reference point when process and outcome evaluation 
stages are being couducted. 
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4. Training Materials Selection 

The selection of existing training materials (as opposed to subcontract- 
ing the development of them) is an important activity. The quality of train- 
ing is often governed by the quality of the materials and activities presented 
to trainees. To insure the selection of quality training materials, do the 
following steps: 

(1) Specify: training population - who is to be trained? 

length of training in hours - how long? 

general goals of training - what do you want 
from training? 

type of training - what kind of experience do 
you want for trainees? 

cost - how much can you pay? how much can 
trainee pay? 

(2) Define what you mean by training: 

Do you inte I your training to: yes no 

1. present information 

2. develop awareness 

3. communicate theory /knowledge 

4. promote identification and application 

of principles 

5. develop an orientation 

6. change attitudes 

7. produce skilled behavior 
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(3) Examine and judge the goals and objectives of the training 
package in terms of: 

acceptable unacceptable 
to you to you 

1, correspondence with your objectives 

2, course length 

3, relevance to your needs 

4, scope of objectives 

5, behavioral quality of objectives 

6, cost per trainee 

(4) Examine and judge the experiences provided for trainees: 

yes no 

1. Are they related to your objectives? 

2. Do they require active participation 
of trainees? 

3, Are experiences varied? 

4, Are they likely to be favorably received 

by trainees? 

(5) Examine and judge built-in evaluation devices: 

1, Are trainees made explicitly aware of 
outcomes? 

2, Are trainees given opportunity to evaluate 

their own knowledge of skill acquisition? 

3, Are trainees provided feedback on their 
experience? 

4, Do trainees have opportunity to rate 
effectiveness of training? 
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(6) Examine and judge background information on training program: 

yes no 

1. Do developers have adequate qualifications? 

2. Is instructor's role clearly specified? 

3. Is previous evaluation data presented? 

(7) Summarize your judgments and decide: 

1. Use program as is. 

2. Seek modifications in the program to make it 
more congruent with your needs. 

3. Judge it as inadequate for your needs. 
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5, Evaiuation design and Instnanentaticm 

This activity is a critical part of the training evaluation mode, input 
stage. In all probability, this should be a major responsibility of the 
evaluation subcontractor - unless you have in-house capability. The steps to 
be followed closely parallel the guidelines for subcontracting evaluation, 
presented in Section IV. 

(1) Specify the type of evaluation design to be used (e.g., 
pre-post test, control group design; post cast only, 
control group design) . 

(2) (a) List the evaluation instruments to be used 

from commercial sources and/or 
(b) State the type of evaluation instrument to be 
developed (e.g., multiple choice items; mini- 
case study; forced choice items). 

(3) Specify the type of analyses to be performed. 

(4) Specify the form in which the data are to be reported 
(e.g.. Decision Information Summary). 

Notes : 

A. The development of evaluation items is both a demanding and a 
creative task. To gain some idea of the type of items that can be developed, 
a perxisal of the following sources is recommended: 

Bloom, B. S., et al. Handbook on formative and surmxtive evaluation of 
student learning. New York: McGraw-Hill, 1971. 
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Bloom, B. S., et al. Taxonomy of educational objectivee : cognitive 
domatn. New York: David McKay Co., Inc., 1956, 

Gronlund, N. E. ConstivACting achievement tests. Englewood Cliffs New 
Jersey; Prentice-Hall, Inc., 1968, ' 

Krathwohl, D. R. , et al. Taxonomy of educational objectives: affective 
domain. New York: David McKay Co., Inc., 1964, 

Thomdike, R. L. (Edo) Educational mcoBuvement. (2nd Fd.) Washington, 
D.C: American Council on Education, 1971, 

B. Testing as part of a training program is generally perceived to be 

a threat by most adults. Usually the threat arises from fear of being rated, 

or fear of not performing well on the test. This threat factor can be 

greatly minimized by using the following techniques (though grumbling should 

still be expected) : 

(1) Stress the fact that information is being gathered to 
evaluate the effectiveness of the course - not to rate 
or grade the trainees. 

(2) Assure anonymity of response by not requiring names on 
papers; if more than one administration is required, 
use a coding system. If coding is used have one of the 
trainees keep the master code sheet and destroy it at 
the end of training. 

(3) Humor the trainees through the "test"; they will usually 
take it in stride if presented properly. 

C. The threat of testing can also be minimized by developing test 
items that :an be used as part of instructional sequences. For exanq)le, 
case studies could be presented with multiple choice answers. The trainee 
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responses could be tabulated and recorded orally, providing group data for 
the evaluator, and then the item could be used for discussion purposes. 
Gathering data of this type would be helpful in providing feedback to instruc- 
tors and developers; exercises of this type could also prepare the trainee 
for end-of-course testing by giving him a chance to practice on items 
similar to the criterion items. 
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ff. Coneietenoy Evaluation 

This evaluation activity is conducted by an observer when training Is 
in operation^ The purpose is to ascertain that the "paper plans" and the 
actual operation of training are the same. Lack of correspondence of train- 
ing in action with information gathered in the training evaluation input stage 
will likely invalidate the entire evaluation (as well as training) process. 
The following steps should be followed by the observer: 

(1) Place yourself unobtrusively, yet strategically, in the 
training environment where you can view the trainees 
and the instructor. 

(2) Place open in front of you for easy reference copies of 
the objectives for training, instructor's manual, 
trainee's material, and contextual assessment informa- 
tion (from Training Evaluation Input No. 3). 

(3) During the instruction, make extensive notes on the ac- 
tivities being conducted, actions taken by the instruc- 
tor, responses given by the trainees. Do not make in-- 
ferenoes at this point; merely keep a running record of 
the action taking place. 

(4) During the first long break (usually the lunch hour) do 
the following: 

(a) Review your notes and eummarize the types of 
learning activitiee engaged in by trainees and 
instructors - do not iapose judgments at this point. 



EKLC 



29 



(b) Review your notes and summarize the most frequently 
observed behaviors of both trainees and instructors, 

(c) Review all the contextual assessment information 
gathered prior to training. 

(d) Now make tentative judgments about the consistency 

of your observations with the objectives, pre-specif led 
training activities, and pre-specified role expecta- 
tions (behaviors) for trainees and instructors. 

(5) If you judge your observations to be consistent with the pre-specified 
information, tht proceed to Step 7 of the Training Evaluation 
Process Stage, 

(6) If you judge that your observations are inconsistent with the 
pre-specified information, then do some or all of the following: 

(a) Select no more than two of the most serious dis- 
crepancies you have observed. 

(b) Take extensive, descriptive notes of behaviors, (actions, 
reactions) during the next session including as much 
actual record of verbal behavior as possible. 

(c) Generate hypotheses as to why the lack of correspondence 
exists and gather information as in (6) (b) to verify or 
negate your hjrpotheses. 

(d) Talk informally with trainees about the training activity 
and ask questions - what do you enjoy most about the train- 
ing? What ways do you think the training might be improved? 
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(e) Talk with the instructor and ask him questions about the 
trainee group, the way things are going, the extent to 
which his expectations for the course are being fulfilled. 

(f) If during informal discussion with the instructor he does 
not refer to the discrepancies that you have observed, 
bring the point up in discussion. Present your informa- 
tion diplomatically and ask for his interpretation of your 
information. State the assuii5)tion that you may have erred 
in your observations and ask him to straighten you out. 

do otherwise will guarantee that you will become a 
pereona nan giKLta and cut off the instructor as a source 
of information, 

(7) Summarize your findings from all of the above and f. a for the 
record. 
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7. BehaoiovaX Observation and Feedback 

Behavioral observation and feedback can be formal, informal or both. 
Formal observation usually requires the use of valid observation system with 
trained observers /recorders, Infomnal observation assumes that a program is 
being implemented as planned (no serious discrepancies identified); its pur- 
pose is to provide confirmatory feedback or corrective feedback to instruc- 
tors, trainees or program developers. Unobtrusively gathered information may 
also provide valuable data for program iin)rovement. Steps to be eiH)loyed are 
as follows: 

(1) If there is a compelling reason to use formal observation 
systems and trained observers, then - 

decide If you want 

(a) a trainee-instructor interaction system or 

(b) a task observation rating scale. 

(2) If a behavioral interaction system is required then consult 
the following source and select the most relevant system; 
train personnel to lise it: 

Simon, A. & Boyer, E, G. MirvorB for behavior II j 
an anthology of observation inatrumente. Phila- 
delphia: Research for Better Schools, 1970. 

(3) Use dome of the Informal Information procedures suggested 
under consistency evaluation, 

(4) Develop behavioral task list to use as a means of recording 
the frequency with which behaviors are observed. (See Job 
Performance Measurement mode for specific exan^les of job 
racing tasks.) 
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8. Training Outoome - Pevformanae 

Outcomes of training presumably reflect the objectives stated for 
training listed in the "Input Stage". The four levels of outcomes specified 
in the DOES chart may occur in any one training program. Where skill train- 
ing is involved, this type of evaluation is the most in?)ortant. 

Steps for inqjlementing the performance outcomes follow: 

(1) Confirm or deny the Input evaluation activity that the 
Behavioral outcomes and instruments specified are still 
relevant performance outcomes following your observation 
of training, 

(2) Confirm the Process stage evaluation activities that: 

(a) training was indeed directed toward the criterion 
performance objectives contained in the evaluation 
instrument; 

(b) trainees were actually given the opportunity to 
practice the tyipes of performance bahavlor required 
by the evaluation instruments. 

(Note: If you are unable to confirm the above, the out- 
come evaluation wi'' 1 probably be invalid.) 

(3) Review the conditions and directions specified for test adminis- 
tration to insure that it is being properly administered. 
(Failure to insure this may Invalidate results.) 

(4) Collect, analyze and report the data; formulate decisions. 
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9. Training Outcoma - Knowledge 

Knowledge outcomes are legitimate training outcomes, particularly when 
terminology or concepts are required to communicate within the field. The 
nature of training may require that the trainees deal with abstract problems 
as opposed to real world problems. 

Steps for knowledge outcomes: 

(1) Check out behavioral outcomes, training materials and evalua- 
tion instruments to ascertain what the knowledge outcomes 
are, how they are to be presented and tested. 

(2) Review key terms in behavioral specification - (e.g., under- 
stand, paraphrase, apply, and con^are them aare fully with 
the test items). 

(3) Review Bloom's Taxonomy of educational objeativea: cogni-- 
tive domain (cited elsewhere) to verify the level of cog- 
nitive functioning as being appropriate to the key words 
used in the behavioral objectives; also, check the model 
test items presented by Bloom against the items to be used. 

(4) Review notes and information from process evaluation acti- 
vities to insure that terms, concepts, principles, have 
actually been presented. 

(5) If time permits and a subject is willing, go over test 
items with the subject to gain an understanding of how 
items were interpreted and the reasoning process used to 
arrive at an answer. 
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10. Training Outcom - Awca^^eneaa 

Some "training" courses may have as their only objective to increase 
the level of awareness of the trainee or provide him witli information. These 
courses are usually of short duration and if they are judged to be worthy of 
evaluation, then these steps should be followed: 

(1) Review the input stage information to verify the type 
and level of awareness desired as a function of training 
as expressed in: needs assessment, behavioral outcomes, 
evaluation instrumentation. 

(2) Review process stage information to confirm the consis- 
tency among goals, objectives, training activities and 
evaluation items. 

(3) Observe "test" administration to insure conformity 
with prearranged conditions. 

(4) Gather unobtrusive information, (e.g., nvmber and type 
of comments about training). 

(5) Process and report data. 
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II. Training Outcomes - Affective 

Often equally important to performance and knowledge outcomes of a 
training course are the affective "feelings" it produces in trainees. If a 
course is designed to make trainees become more confident in their job per- 
formance, as well as more competent, then it is inq)ortant to attempt to 
measure these outcomes. The following steps pertain: 

(1) Review the affective behavioral ou comee for the 
training course. 

(2) Review the affective evaluation instruioBnts or items to 
determine congruence with objectives. 

(3) Review process stage data for any evidence of direct 

or unobtrusive information related to affective outcomes. 

(4) Conduct interviews with selected trainees and try to 
gather, directly or Indirectly, information related to 
affective outcomes. 

(5) Review data from test Instruments. 

Note: 

Interviews may result in unreliable information. Trainees may tend 
to tell you what they think you want to hear. If they have been released 
from their normal work load to attend the course, they are likely to feel 
positively about it - unless they have been soundly turned off. Why should 
they "knock" a good thing? Consequently the evaluator may have to employ 
more "creative ways" of gathering this data than through interviews or 
ques t ionnalres • 
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SECTION III, OPERATIONALIZING 
THE DOES SYSTEM 

Part B: Job Performance Evaluation 

Aotivitiea 

Input stage I. Job Analysis and Description 

2. Task Inventory 

3. Task Analysis 

4. Feedback of Task Analysis Information 
to Training Development 

5. Development of Perforrrance Measures 

6. Specification of Performance Criteria 

7. Observer Selection and Training 

process stage 8. Conaistenay Evaliuxtion 

9. Observation of Performance and Feedback 

outcome stage 10. Training Level Outcome - Performance 

11. Training Level Outcome - Affective 

12. Formative Evaluation of Performance 
Measuring Instruments 

12. Recomnendations for Decisions 
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I* Job Analyaia and DeeaHption 

Job analysis and description Involves the identification of performance 
requi rements for an occupation. The steps are: 

(-.) Collect and review sources of job performance data. 

(a) Organizational charts 

(b) Dictionary of Occupational Titles (UoS. Department of 
Labor, Superintendent of Documents, U.S. Government 
Printing Office, Washington, D.C.) 

(c) Job pe ^ormance literature, e.g., handbooks, 
manuals, bulletins 

(d) Interviews 

(e) Questionnaires 

(f) Materials and equipment used in performing work 

(g) Observation of performance 

(h) Performance of the job by the analyst 

(2) For the particular job under consideration set down a list 
of tentative descriptions for that job. 
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2. Taek Inventory 

The task inventory is a procedure which identifies specific tasks performed 
on the job for each job category. The six steps for task inventory follow: 

(1) Review job descriptions^ training packages , catalogs and 
other standard references • 

(2) Survey of supervisors and other job experts (interviews 
and/or questionnaires) • 

(3) Surveys of job incumbents (interviews and/or questionnaires), 

(4) Behavioral observations, 

(5) Performance of tasks by the analyst or evaluator. 

(6) Formulation of behavior-referenced task statements according 
to the following criteria: 

(a) Use action verbs (e.g., "troubleshoots refrigerator 
thermostatic control", not "refrigeration maintenance"). 

(b) Use clear, unambiguous language (e.g., "repairs leaky 
gaskets in faucets", not "maintains plumbing"). 

(c) Use "time rateable" statements to indicate a clear 
idea of how much time it takes to perform a task. 

(d) Task statements should, insofar as possible, be 
mutually exclusive. 

(e) The list of task statements should be reasonably 
exhaustive for the particular job. 

(f) Tasks should be realistic (Morsh & Archer, September 1967). 
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3. Task Analyeie 

The procedure of selecting and categorizing tasks to form the basis for 
training objectives and job performance measurement is called task analysis. 
Steps which implen^nt this procedure are: 

(1) From task inventory prepare a rating instrument that 
ascertains for each task its 

(a) realism 

(b) time rating 

(c) importance 

(d) source of training. 

(2) Specify criteria for salection of task items on the basis 
of ratings listed in Step 1. 

(3) (Pilot test). Administer task inventory instrument to 
panel of experts (e.g., first-line supervisory and a 
sample of job Incumbents. Call for additional task item 
to be listed. 

(4) Perform statistical analysis of task ratings to select 
tacks meeting criteria specified in Step 2. 

(5) On the basis of Steps 3 and 4 revise task inventory and 
administer to large sa^>le of Job incuaibents. 

(6) Repeat Step 4 and revise instrument if necessary. 

(7) Categorize task into functional groupings based on common 
behavioral elements; e.g., communication, troubleshooting, 
repair, supervisory tasks, etc. 
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4. Feedback of Task Analysis 
Information to Training Development 

Task analysis infoncatlon la fed back Into development of training so 
that desired behavioral outcomes of training can be specified. The steps 
ares 

(1) Review comnonallty among tasks and functional groupings 
to facilitate development and organization of course 
materials, and to 

(2) Translate into training objectives. 
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5. Development of Performance Meaaux-ee 

Development of performance measures, i.e., performance tests, per- 
formance rating scales, and critical incident logs is the procedure which 
provides task-derived instruments for measuring transfer of training to 
job perfoz^ance. 

(1) Where feasible, on the basis of cost, availability of 
equipment, and availability of "trainees" time, develop 
performance tests within these functional groupings of 
tasks (biggest payoff comes from being able to represent 
large task- groupings with a single, generalize able per- 
formance test - probably never achievable). 
Note: Make the items in the performance test corres- 
pond to the tasks performed on the job (Fredrickson, 
1962; Wilson, 1953, 1962). 
(2) Where development of performance testa ie not feasible, 
develop a performance check- list or performance rating 
scale based on task items determined from task analysis. 
This is done to make the rating methodology task-specific. 
The scale should tap individual competency and proficiency 
in performance of tasks and lead to diagnosis of indi- 
vidual training needs for both formal course training and 
on-the-job training (Wilson, 1962). 
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6. Specification of Pf^vformance Criteria 

Performance cxrlteria must be specified to provide standards for 
evaluating performance maasures. To do this: 

(1) Have panel of experts rank order or assign weights to 
performance measurement items. 

(2) Have panel of experts establish cut-off scores for 

each item as criteria of successful training* 
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7. ObaeTver SeleoHon and Training 

To Insure that observers have knowledge of task performance skills 
and can use the performance measuring Instrument, follow these steps: 

(1) Select ob8er\'ers who have experience In performing 
Incumbents' tasks and who have opportunities to ob- 
serve Incumbents at work (e.g., first-line super- 
visors) . 

(2) Instruct and train observers in use of performance 
measuring Instruments. Give observers skill practice 
with instrument under supervision of evaluator. 
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5. Conei&tenoy Evalmtion 

Consistency evaluation helps determine the degree of cong 'between 

measurable tasks and training objectives and experiences. ( ais oc dure 
is analogous to Training Process Evaluation Procedure No, 6). T]ie ^steps 
are: 

(1) Review curriculum objectives and lesson plans to 
ascertain whether specific job tasks are included 
in training program, 

(2) Observe instructional process to ascertain inclu- 
sion of specific job tasks, 

(3) Interview trainees to determine whether instruction 
was received on specific job tasks. 
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9, Obeervation of PerfoT nance and Feedback 

This procedure provides developers and evaluators with Information 
the effectiveness of training relative to specific tasks as perfonned 
(analogous to Training Process Evaluation Procedure No. 7). The steps 

(1) P7-ior to training, adnlnlster performance measuring 
inatruments to sanqjle of incuaients. 

(2) Subsequent to training, administer performance 
measuring instrument to sanqjle of trained and 
untrained incun^ents. 
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10, Training Level Outoome - Perfozmanoe 

To determine the extent of transfer of training to job performance 
(analogous to Training Process Evaluation No. 8): 

(1) Analyze results of pre- and post-adoinlatration of 

performance measuring instrument to determine the 

transfer effect of training. 
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II. Training Level Outcome - Affective 

This procedure is to determine the extent to which training has 
Increased confidence of trainees to perform their job. Steps to be fol- 
lowed: 

(1) Analyze pre- and post-adminlstratlon of confidence 
ques tlonnaire. 

(2) Interview job incumbents before and after training. 
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12. Formative Evaluation of 
Perfovmanog MeoBUring Inatnonenta 

This evaluation proctdure xaakes possible the Improvement of measuring 
Instruments. Follo«# these steps: 

(1) Develop design for evaluation of performance measuring 
Instruments, including standards for the instrument. 

(2) Conduct formati\'« evaluation of performance measure- 
ment methods (performance tests, performance check- 
lists, performai?ce rating scales, etc.). 

(3) Revision, evalu^ition, revision cycle until standard 
is met. 

(4) Determine utility of performance measuring instruments 
as criteria for validating training outcome measures. 

(5) Retain performance Instruments that demonstrate accep- 
table utility as criteria for training. 

(6) Use performance measures to produce decision informa- 
tion about training activity. 
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13. Reooimendationa for Dedeione 

This evaluation procedure involves providing feedback of job performance 
information to training developers and sponsors to in^rove quality of 
training services as well as feedback to personnel departments for assess- 
TOnt of specific training needs for particular individuals. 
(1) Apply criteria in Procedure 6 to conq)letion of 
Decision Information Summary for evaluation of 
training activity in job performance mode. Feed- 
back to training developers, personnel and train- 
ing supervisors. 
(2) Feedback to personnel and training supervisors 
information assessing units and individual needs 
for training. 
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SECTION III, OPERATIONALIZING 
THE DOES SYSTEM 



Part C: Community In5>act Observation 

Activities 

input stage I. Corrmmity Needs Assessment 

2. Community Indicator Identification 
and Selection 

3. Impact Analysis Design 



process stage 



4. Monitoring 

5. Unobtrusive Evaluation 



outcome stage 



6. Evalmtion of Trends 

7. Survey Evaluation 
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I. Coimunity Needs Asseesmant 

In order to i'^.vaclfy community needs ^ relevant community agencies 
Inside and ou^seldi f government muse be contacted for lnl:ormatlor», A 
need Is de/.in«?ci at a discrepancy between the current and desired state of 
affairs; community needs, then^ relate to the discrepancy between the 
"quality of life" as it is as it should be in a given community. To con- 
duct a community needs assessment, the following steps are recommended: 

(1) Develop or subcontract the development of a survey 
instrument (questionnaire) to gather Information on 
community needs. 

(2) Identify relevant social and civic organizations and 
governmental units to be used as sources of data in 
conducting the needs aasesement, administer survey 
instrument to each agency. 

(3) Develop a sampling plan for gathering information 
from randomly selected citizens; a'^minister survey 
instrument or conduct structured interviews. 

(4) Develop a plan for setting priorities among the needs 
identified by the various groups to whom the survey 
instrument is administered. (You may consider use of 
the Delphi Technique in achieving this priority setting. 
See Cook, D. S.). 

(5) Set priorities among the various needs identified. 
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(6) When priorities have been established, review the list 
to identify the type and number of needs that you judge 
can be satisfied by training of municipal employees, 

(7) Review the list generated by the above process and for 
each need: 

(a) Estimate the cost of training required to meet 
the need (e.g., dollar cost per trainee), 

(b) Estimate the penalty of ignoring the need (e.g,, 
what is likely to happen if you ignore it), 

(c) Classify training activities on a cost-benefit 
continuum (e,g., the cost of training street 
maintenance personnel is low, the direct commu- 
nity benefits are high, the cost of training 
supervisors is high, the short range direct 
community benefit is low) • 

(8) Convene a panel of consultants knowledgeable in municipal 
affairs and ask them to critique your needs assessment. 

(9) Revise community needs assessment and pub.lish, (Note: 

A needs assessment is never complete. It must constantly 
be updated to reflect emerging conditions,) 
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2. Community Indicator Identification and Selection 

Th« problem of evaluating community impact can probably best be handled 
through the gathering and processing of natural data on community life. In 
this respect, it is important that accurate records be maintained and made 
acces3ible. To identify and select indicators: 

(1) Ask all municipal departments, community social, civic 
and recreational agencies (and state agencies where appli- 
cable) to provide a list of the type of records they keep. 

(2) Ask for samples of the records kept. 

(3) Con?)ile a list of the various types of data available from 
each of the agencies and classify each type as unrelated 
or related in some way to a community need. 

(4) Re-examine the list of data types and classify them as being 
directly related to a need identified in the community needs 
assessment. (Note: Citizen con5)lalnts may provide the best 
indicators of the quality of life in a community. Whil^ in- 
direct, albeit negative Indicators, citizen con5)laints are 
likely to bear a higji degree of relationship to needs identified.) 

(5) Decide on a list of Indicators for which accurate data will 
be kept as a means of evaluating txnpazt. 

(6) Institute procedures, where necessary, to standardize the 
format of data collection and presentation so that later 
analysis is made easier. (Note: See appendix in the 
Technical Report to view the public service system developed 
by Mayor Hubbard of Deaibom, Michigan.) 
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o, Impaot Analysis Design 

This activity of the Input Stage requires a technical expert in 
research/evaxuation design. The activity involves the following steps 

(1) Review the information gathered in previous input 
stage activities; refine the research/evaluation 
questions. 

(2) Specify the data to be collected, when it is to be 
collected, and the method /conditions of collection. 

(3) Specify the analysis and reporting procedures to 
be employed. 
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4» Monitoring 

The purpose of this activity is to insure that data collection pro- 
cedures are being carried out as planned. The following activities are 
included: 

(1) Review the information from all sources in the Community 
Iiiq)act Observation Input Stage, 

(2) Review the data as they come in to insure: 

(a) that data are being reported accurately and 
in useful form; 

(b) that there are no missing data; 

(c) that there is conformity to prespecified data 
collection decislGas; 

(d) that "rroubl^: ' Is r: . . brewing. 

(3) Review dat^ to ^ 3 if trendu are developing. If they 
are not developing on uijy want to question your 
assuii5>tions, procedures and hjrpothesize why the data 
are as they are. (Not- : laiyact evaluation is a 
very new field, fraught with conceptual and metho- 
dological problems - don't except too muchi) 
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5. UnobtrueiDe Evaluation 

So^. 1 of the most powerful data on conrraunity impact as a function of 
training may come from unrelated sources. To remain alert to these poten- 
tial sources of influence we recommend the following steps: 

(1) Personnel in government (sponsoring) agencies, trainees, 
evaluators should brainstorm potential sources of data 
(as a source for brainatorming techniques you may wish 
to use Fames, S. Creatit>e Behccvior Workbook. New 
York: Charles Scribners,, 1967. 

(2) Read Webb, E. J., et al. Unobtrusive measure: non- 
reactive research in the social Boienaee. Chicago: 
Rand McNally, 1966. 

(3) Analyze the information gathered from Activities 1 
and 2 in this section aad narrow the range of choice;. 

(4) Based on the above, decide if it is feasible to col- 
lect some unr Lrusive data; if so develop and imple- 
ment a design. 



EKLC 



57 



6. Evaluation of Trenda 

Analyses of naturalistic data on community life (e.g., police records, 
unemployment, service delivery records) provide the basis for evaluation 
of trends. It should be noted that it may take a year or more to collect 
and analyze data for this type of evaluation. To be useful, naturalistic 
data should be carefully recorded and easily retrievable. Evaluation of 
trends requires the following steps: 

(1) Review all information from Community Impact Observation 
Input and Process activities to "posture" yourself for 
the evaluation. 

(2) Pay particular attention, in your review, to the quality 
of data collected during the process stage; are they 
"fine enough" to detect differences (e.g., are data 
recorded in workable units of time sufficient to detect 
shifts in frequency or response time?). 

(3) Look for some measurable attributes of the effect of 
training in your data (e.g., reduction of mean down 
time of machinery, reduction in number of complaints 
about police, reduction in mean repair time required to 
fill pot holes In city streets). 

(4) Reanalyze the research/evaluation question and derlgn in 
the input stage and question its adequacy with respect 
to the quality and quantity of the data you have. Re- 
design the analyses if the conditions of the data warrant 
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it* Be prepared to abandon data if they shed no light 
oxi the questions of interest. 

(5) there appear to be "effects" in the data that are 
unaccounted for by your questions or analyses: 

(^) generate new hypotheses for the occurence 
of changes detected; 

(b) design and perform sequential analyses to 
account for the data* 

(6) I/Oolc for patterns in the data as potential assignable 
''Causes" for the observed trends in the data. 

(7) continue the evaluation of trends until you are s^itis- 
fted that you are: 

(a) asking the "right questions"; 

(t>) getting reasonable, economical answers; 

(c) getting "good data"; achiev ag stable change; 

(d) detecting assignable causes that lead to 
the generation of new training programs. 
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7. Sicrvey Evaluation 

There are times when w designed questionnaires and well conducted 
Interviews are highly desirable forms of gathering data on community im- 
pact. Deficiencies In these tjrpes of data collection are treated In the 
pc jltlon paper on Community In5)act Evaluation. If you decide to use 
questionnaires/Interviews, do the following: 

(1) Review the Information collected from the Input Stage 
and Process Stage: 

(a) pay special attention to the community Indicators 
Identification; 

(b) review th^. procejfli stage data (where applicable). 

(2) Review all avaluaclve data from all stages of both the 
Training Evaluation and Job Performance Measurement: 

(a) be sure you know Wactt the ob^ectii^s of training 
were and wha*: the outcomes of trainlv.j evi-luation 
were; 

(b) rev5.?2w the Job Performanv^G Measurement Information 
from all three stages* 

(3) Decide oa whi'.ch form (questlorinnlre/iater\'le*^) you will 
use and what the fo7UB will be. 

(4) If you decide on a questiornalre, upe the following step.>: 
(a) make a clear^ concise statement of the iaforma- 

tion you are seeking; 
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(b) decide on the type of questionnaire you will use 
(e.g., open-ended, forced response); 

(c) develop a first draft of the questionnaire; 

(d) re-examine the questions with reference to Items 
1-3 in this general category; 

(e) pre-test the questionnaire on a sample of the 
population; 

(f) revise based on pre-test; 

(g) prepare administration procedures for the 
ques tioiinaire • 

Good sources for help in questionnaire construction include 
the following: 

Komhauser, A, & Sheatsley, P. B. Appendix C. Ques- 
tionnaire Construction and Interview Procedure, in 
Selltiz, C, et al. Recearah Methods in Social 
Relations. New York: Holt, Rinehart & Winston, 
1959 » 

Oppenheim, A. H. Questionnaire Design and Attitude 
Measurement. New York: Basic Books, 1966. 

(5) If you decide to use an interview procedure, follow the same 

steps in construction as presented in Item 4. In addition 

to the reference sources mentioned above. Chapter 7 in 

Selltiz, 1959, is recommended for reading (Chapter 7, 

Data Collection II: Questionnaires and Interviews). 

A note on oormunity impact evaluation: 

IxnpRct evaluation measures have, for the most 
part, been developed for projects or programs which 
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can define a "closed" target population as in military 
units or business institutions. In publicly adminis- 
tered programs, however, adminis tr£*tors do not have 
direct access to in5)act information. This situation 
requires the evaluator to rely on "natural process 
records" (routine data collected for another purpose: 
e.g., city complaint files) for possible indicators 
of impact of training. In such cases, the evaluator 
must have a working knaoledge of corrrmmity and train^ 
ing objectives as derived from Needs Assessment. 
Using these objectives as a yardstick, he caii measure 
community impact against such natural process data as 
can be obtained. 

Although a community needs acsessoent can focus 
on specifically observable phenomena, it is important 
to understand that "social indicators" offer some 
considerations for decision-makers, Henriot (1972) 
suggests that: 

**soQial indicators are quantitative data 
that serve as maaeures of socially important 
conditions of society. These indicators may 
measure both 'objective' conditions of 
society and persons (e.g., health, educa- 
tion, crime, mobility, etc.) and 'subjec- 
tive' perceptions of life experiences 
(e.g., satisfactions, aspirations, aliena- 
tion, etc.) (p. 3)." 

More importantly, however, it is pointed out 
that social indicators "...are basically a matter 
of values, of interest, of policies - hence, of 
politics (Henriot, p. 5)." The basis for this 
statement is provided by the following points: 

"1. The concern for 'quality of life* is 
a highly political concern; hence measure- 
ment of the quality of life (a practical 
objective of social indicators work) haj3 
inevitable political implications (t>.5). 

2. The commitment to social indicators work 
springs from an identifiable political posi'^ 
tion (pp. 6-7). 

3. The generation and utilization of social 
indicators takes place in a thoroughly poli'^ 
tieal environment (p. 7)." 

And it is concluded that "It is simply not 
possible - or desirable - to isolate the social indi- 
cators task from political influences (p. 8)." 
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Figure 3. Supporting Components of DOES 
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SECTION IV 



SUPPORTING The activities which operationalize DOES have been 

COMPONENTS 

detailed in the preceding section. This section 
presents important eupportivs proaeduree as follows: 

1. Recommendaticne for Search and Analyaia Proaeduree 

2. Hew to State Goals and Objectives for Training 

3. Procedures for Securing BFP's for Training 
Materials Development 

4. How to Sul^contraat Evaluation Studies 

5. A Guide for Non^Evaluators : How to Judge 
the Quality of an Evaluation Study 

6r How to Use the DOES Reporting System 
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I. Reoomnendationa fox* Search and Andlyain Proceduree 

(1) Ideally, the first stage of an application-oriented search and analysis 
effort should include a search for materials in the immediate community. 
These materials will provide a basis for general reading in the subject area. 
Frequently, community agencies are able to offer background information and 
help place otherwise isolated data into context. 

(2) The second stage ie to establish analysis criteria of acceptable infor- 
mation or a general outline based on the initial collection of material. 
This should include not only Information from the immediate community, but 
also examination of such general reference sources as encyclopedias in rele- 
vant fields, annual reviews of literature, and so forth. If an outline is 
used, it should be broad enough to absorb new material and data which are 
collectecl at a later date. 

(3) The next stage is to search for material within specifically defined 
areas of inquiry (based on the outline/criteria established). Information 
should be collected from the following sources: 

(a) interviews and site visits (first within the immediate 
area and then elsewhere) ; 

(b) con?)uter searches for literature (in some fields, basic 
research information is indexed in machine-readable form 
and stored on computer tapes); 

(c) key journals should be searched individually, issue by 
issue, for at least the last five years; 
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Cd) bibliographies and reference information from journal articles 
should be continually searched and recorded; 

(e) microfiche should be ordered when possible, since it reduces 
the purchase price of documents considerably; 

(f) information should be continuously solicited through corres- 
pondance, since much of the most recent information is not 
obtainable in city and university librar>' systems; 

(g) books covering the general area should be searched for 
specific information which satisfies analysis criteria or 
outline information neede; 

(h) specialized reference sources such as Psychological Abetraote 
should be systematically searched. 

As material arrives, it should be handled in the following manner: 

(a) All material should be put in some presentable form and 
photocopied. If abstracts or other information taking less 
than a full page are collected, they should be cut and pasted 
on a sheet of full typing paper. The originals :hen be 
filed alphabetically. From this point on, only the copies 
are handled, the originals are now in their final flies, 

(b) Reference notecards in American Psychological Association 
form should be filed in a file box in alphabetical order. 
Copies of these cards should be attached to each copy of 
reference material. It should be emphasized that complete 
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reference information must be collected when the original 
reference is first retrieved, 

(c) The reference materials (copies) should be categorized accord- 
ing to the established outline/criteria as they are collected. 

(d) The reference material should be regularly distributed to 
individuals requesting information for «ftevaluation and 
categorization. 

(e) Each reference card should be marked stating the category in 
which the material has been placed. 

(f) A method for the organization of information: 

(1) Underline key passages of material or abstract 
the material. 

(2) In order to make these abstracts or underlining avail- 
able to all members of a project (to cut down on their 
reading time and to facilitate any writing task) a 
master outline should be gradually developed by plac- 
ing these passages ia the appropriate places within 
the oui a or under the criteria headings which they 
support, 

(3) In order to keep accurate reference information, it 

is necessary to include in the margins of such passages^of T^^ >vh ^/c* 
'f\^-hvvv>^ the author's name, year of publication, the page num- 
ber of the statement, and the first words of the title 
(if the author has more than one publication in that 
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particular year) • 
(4) Each piece of reference information should be placed on 
a separate page with the appropriate reference infor- 
mation included. These should be arranged in the order 
of the outline and copied. The originals should be 
filed and the copies bound, 
(g) A card catalogue for references is recommended: 

(1) It provides a convenient means by which to check material 
out to a project member by writing the project menier's 
name on the back of the card along with the date, (When 
the material is returned and filed, the name is crossed 
off J 

(2) It is a heipful aid in typing the final bibliography 
and reference list of any writing effort. Since the 
cards are already in the correct form and alphabetically 
arranged, a typist can compile a reference list by 
tnerely typing cards in the order and form in which 

they appear, 

(h) Tlie copies of the cards which are attached to the copies of the 
reference materials are usefu], in that a project member can 
cite his source more readily. This avoids much reference con- 
fusion. 

(5) Ther should be regular staff meetings to reevaluate the established 
criteria or revise and make additions to the outline. 

(6) The final product of search and analysis leading to some form of vrritten 
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material should consist of the following: 

(a) a file of the original references arranged alphabetically 
by author; 

(b) a file box of reference cards, written in the final form, 
also stating the particular category under which copies 
of the material have been categorized and filed. These 
cards should be arranged alphabetically. 

(c) All copies of material, filed according to particular 

ertablished categories > or cut and pa3ted^« underliaed ttw^ iA^cAtf'Ji , 

^ outline form (these should then be bound into a series 
of volumes and paginated for easy reference). 
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2. Hod to State GoaU and Objectivea for Training 

(1) Goals : 

The goals of training should be derived from needs assessment data 
and to some degree, job analyses information. Goals are statements related 
to the area for which training activity is to be developed. The best goal 
statements are those which specify an observable outcome, the conditions 
under which this outcocje is to be observed, and a standard against which 
the observable outcome may be compared. The following is an example of a 
good goal statement: 

"Given a six-month period (after training) the number of com- 
plaints about electrical service will be 50% of the number of 
complaints registered in the previous six-month period." 
Notice that a goal statement indicates only what is to be accomplished and 
not hou) it is to be accon^lished. A "six-month period" is the condition, 
the "number of complaints" is the observable outcome, and "50% of the number 
of complaints previously registered" is the standard. The area of training 
is electrical service. The sequence of specifying training goals, therefore, 
is: 

(a) Identify and specify an area of training; 

(b) specify an observable outcome; 

(c) specify the con.i ., as under which this outcome is 
to be observed; 

(d) specify a standard for evaluation. 
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Kaufman (1972) presents some alternatives to developing instructional 
goals. See also: Browder, L. H. , Jr. Emerging Patterns of Administrative 
Accountability, Berkeley, California: McCutchan Publishing Corporation, 1971. 
(2) Objectives: 

There are two levels of objectives which may be stated with regard to 
training materials development (Grcalund, 1970): (l) An instructional 
objective is c general outcome of instruction, and (2) a behavioral objective 
is a specific behavioral outcome of training. The major differeace between 
instructional objectives and behavioral objectives is one of order. For example, 
an instructional objective might be: the trainee will have an understanding 
of supervision principles. Behavioral objectives, then, will be specified 
as evidence of attaining the instructional objective. Thus, in this instance, 
behavioral objectives might be: 

(a) Given a list of 30 statements, C0% of the trainees will be 
able to select correctly 80% of those statements which are 
basic principles of supervision; 

(b) Given a list of 10 principles of supervision, 80% of the 
trainees will be able to write an adequate description of 
the supervisor's role for 8 of the principles. 

Some of the sources which would be helpful in writing behavioral outcomes are: 

Mager, R. F. & Beach, K. M. , Jr. Developing Vocational In- 
etruation. Palo Alto, California: Fearon Publishers, 1967. 

Gronlund, N. E. Stating Behmnoral Objectives for Classroom 
Instruction. New York: The Macmillian Company, 1970. 

Gregg/McGraw-Hill & The Center for Vocational and Techr-'cal 
Education at The Ohio State University. Writing Performance 
Goals: Strategy and Prototypea, New York: Gregg Division/ 
McGraw-Hill Book Company. 
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3. Prooedurea for Securing RFP'e 
for Training Materials Development 

(1) Identify potential clients 

(a) Survey colleges and universities to identify departmenns, 
agencies, service bureaus, and individuals that have exper- 
tise in your area of interest, 

(b) Survey professional organizations related to the general 
field and identify a contact person, 

(c) Survey related governmental agencies (state and federal) for 
namas of individuals who could provide you with information. 

(d) Survey Cormeroe Bmineas Daily (U, S, Government publication) 
for best of grant awards to private research and deveiopineat 
firms doing business in your area of interest, 

(e) Collate and classify information gathered above for reference, 
(2) Prepare a "Letter of Interest" 

(a) Prepare a letter of interest for institutions listed above to 
ascertain their interest and availability to provide the ser- 
vices you are requesting, 

(b) In the letter of interest provide the potential client with: 

(1) brief statement of purpose of proposed project; 

(2) scope and length of the proposed contract; 

(3) general requirements and constraints imposed on 
client (e.g,, likely amount of the contract; 
conditions that must be met). 
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(c) Aak Client, if Interested in project, to provide you with a 
brief prospectus of his organization including: 
(l) actual work done related to contract area; 
^2) list of personnel (w/qualifications) who are likely 
to be assigned to the project if contract is awarded. 
(3) Screed Letters of Interest 

(a) Individuals and organizations who have little or no exper- 
ience in your area of interest should be screened out at 
this point. 

(b) goine individuals and groups may not have proper qualifi-- 
cations for the work and should be screened out. 

Note: Tlie ^i^velopment of a response to an RFP id often a costly experience 
for both client and the sponsoring agency; careful screening at this stage 
can save ali parties considerable money. 

(c) select those individuals and/or agencies who are most 
interested and best qualified to do your work. 
(4) Prepare the RFP* 

(a) specify the training problem in one or two sentences. 

(b) Djrovide information on the education, age and work background 
of the training population. 

(c) St^te what you would like the trainee to be able to do at 
th^ end of training. 

* Be sure tl^^t you include a statement indicating that you reserve the 
right to jr^ject any or all bids. 



EKLC 



73 



specify the work task for the bidder. 

(1) Specify the product you expacv from the bidJer and 
its characteristics (e.g., the final product will 
be a set of training aaterJalH and acco.fl.\:anying 
leader* s guide designed to train first line super- 
visors in municipal government to deal with the 
handling of minority group grievances, Tlie train-- 
ing experiences shall include provisions for role 
playing, problem solving, and small group discussion). 

(2) Specify the delivery date for the product and the 
quantity of the final product, 

(3) Specify che number of field test conditions and 
data reporting (where applicable) . 

Specify the time schedule, budget format, and personnel quali- 
fications : 

(1) Require bidder to provide a time schedule and general 
plan for product development that i.iclude key check 
points for your monitoring purposes. 

(2) Provide budgat format you require of bidder, 

(3) Require project staff summary with qualifications 
of key personnel. 

(4) Specify date and time on which bids are to be re- 
ceived and opened r 
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HOW TO RATE CONTRACT PROPOSALS* 



RFP re: 



Date: 



Name of Contractor: 



Total Rating Points 



Items: Contractor Qualifications 

1, Professional qualifications 
Project manager 
Technical support personnel 

2. Experience in training area 
Prior development work 
Related work 

Proposal Qualif ica^tions 

1. Clarity of training objectives 

2. Congruence of objectives with training goals 

3. Congruence of training activities with your 
specifications 

4. Diversification of training activities proposed 

5. Acceptability of training activities to trainee 
population 

6. Provisions for feedback to trainees 

Budget and Time Schedule 

1. Reasonableness of cost proposal 

2. Acceptability of check points in the 
development process 

3. Acceptability of other items 



Total Rating 
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Rate each of the items as follows: 

5 for quality that exceeds expeccation 

4 for highly acceptable quality 

3 for acceptable quality 

2 for marginally acceptable quality 

1 for unacceptable quality 



* Adapted from: Otto, C. P. & Glaser, R. 0. The Management of 

Training. Reading, Mass.: Addison-Wesley, 1970, 



Note: The rating scale is provided as a means to assist in the screening 
of the overall quality of proposals. In the final analysis, one should 
screen very carefully several times, those few proposals that are of tiighest 
quality. Final decisions for selection of a contractor, when there are 
several competing bids, may be such variables aa personal compatibility 
of project officers, distance from sponsoring agency and the like. 
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4, Hau) to Subcontraot Evaluation Studies 

(a) Review the types of evaluation by mode in the DOES evaluation 
system chart. 

(b) Examine Input Evaluation for each mode of evaluation. 

(1) Have these evaluation activities been conducted? 

(2) If not, decide whether you can perform these activities 
yourself or whether you should subcontract them. 

(3) If the evaluation activities have been completed, are 
you satisfied with the results? Can you work with them? 

(c) Make decisions about the types of evaluation studies you 

wish to have performed within each mode (e.g.. Job Per jmance. 
Process Evaluation). 
Identify potential evaluation subcontractors. 

(a) Survey colleges and universities to identify department*? > 
agencies, service bureaus, and individuals that have exper- 
tise in evaluation. 

(b) Survey professional organizations related to the general 
field and identify a contact person (e.g., American Edu- 
cational Research Association^ _ *\ri Management Asso- 
ciation, American Institutes for Research). 

(c) Survey related governmental agencies (state and federal) 
for names of individuals who could provide you with 
potential bidders. 

(d) Survey Cormevce Busineee Daily (U. S. Government publica- 
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tlon) for grant awards to private research and development 
firms who conduct evaluation studies, 
(e) Collate and catalog information gathered above for ready 
re ference, 

(3) Prepare a ''Letter of Interest" to send out. 

(a) Prepare a letter of intere^it for institutions and indi- 
viduals listed above to ascertain their interest and 
availability to provide the evaluation services you 

are requesting. 

(b) In the letter of interest provide the potential client 

wit i: 

(1) brief statement of purpose of the evaluation scudy; 

(2) scope, length of the proposed contract, and expected 
dollars available; 

(3) general requirements and constraints imposed on 
client (e.g., conditions that must be met). 

(c) Ask client, if interested in project, to provide you with a 
brief prospectus cf his organization including: 

(1) actual samples of evaluation work done related to 
contract area; 

(2) list of personnel (w/qualif ications) who are likely 
to be assigned to the project if contract is awarded. 

(4y Screen Letters of Interest 

(a) Individuals and organizations who have little cr no 

expevienoe in your area of irterest should be screened 
out at this point. 
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(b) Some individuals and groups may not have proper qKalifi- 
cations for the work and should be screened out. 

Note: Tite development of a quality proposal response to an RFP is often 
a costly experience for both the client and the sponsoring agency; careful 
screening at this stage can save all parties considerable money. 

(c) Select those individuals and/or agencies who are most 
interested and best qualified to do your work. 

(5) Prepare the RFP 

(a) Specify the evaluation problem in one or two sentences. 

(b) Provide background information on the training population 
(e.g., age» educational background, work experience). 

(c) State the type of evaluation you want conducted by mode 
and by type (e.g., training process and training outcome 
evaluation for X course). 

(d) Provide evaluator with copy of the training materials 
to be used for instruction. 

(e) Specify the work task for the bidder. (The following 
are samples.) 

(i) Indicate the general type of . valuation items you 
want used. 

(a) multiple choice items 

(b) scalable items 

(see Exhibit Y for 

(c) forced choice samples of evaluation 

items) 

(d) case problem, etc. 



79 



74 



(2) Specify timetable for your review of items prior 
to use. 

(3) Specify any pilot test requirements you may have. 

(f) Specify time schedule, budget format, personnel qualifications, 

(6) Send out RFP's, 

(7) Review and rate proposals, 

(8) Decide on subcontractor, 

(9) Negotiate a contract based on proposal, 

(10) Monitor evaluation study in process according to contract, 

(11) Review final reports to determine compliance with original contract. 
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HOW TO RATE EVALUATION PROPOSALS* 

RFP re: Date: 

Name of Contractor: Total Rating Points: 

Items: Contractor Qualifications 

!• Professional qualifications 

Project manager * 
Technical support personnel 

2. Experience la evaluation 

Prior evaluation work 

Related research or development work 

Proposal Qualifications 

1. Clarity of evaluation procedures • 

2. Congruence of evaluation study with training 

goals and oojectives 

3. Congruence of evaluation activities with 
your specifications 

4. Acceptability of evaluation activities to 

trainee population 

5. Provisions for evaluation feedback to trainees, 
instructors and sponsoring agency 



Budget and Time Schedule 

1. Reasonableness of cost proposal 

2. Acceptability of check points in the evaluation 
process 

3. A.cceptability of other items 

Total Rating 
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Rate each of the items as follows: 

5 for quality that exceeds expectation 

4 for highly acceptable quality 

3 for acceptable quality 

2 for marginally acceptable quality 

1 for unacceptable quality 



* Adapted from: Otto, C. P. & Glaser, R. 0. The Management of 

Training. Reading, Mass.: Addisou-Wesley , 1970. 

Note: The rating scale is provided as a means to assist in the screening 
of the overall quality of proposals. In the final analysis, one should 
screen very carefully several times, those few proposals that are of 
highest quality. Final decisions for selection of a contractor, when 
there are several competing bids, may be such variables as personal 
compatibility of project officers, distance from sponsoring agency and 
the like. 
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5. A Guide fov Uon^EvaluatovQ and Hob) vo 
Judge the Quality of an Evaluation Study 

The following set of guidelines are designed for the non-evaluator 
who is required to judge the quality of subcontracted evaluation study. 
It is a belief held by the GATE project staff that a non-evaluator can 
indeed judge most aspects of the quality of an evaluation report. Many 
of the guidelines to be presented will be indirect indices of quality; 
nevertheless, they provide a good basis for judgment. In some cases the 
spor-oring agency may wish to hire another evaluator to audit the quality 
of an evaluation report. 



JUDGMENTS TO BE MADE ABOUT AN EVALUATION REPORT 

Yes No 

(1) Is the report comprehensive; does it cover all aspi^cta 
of training evaluation, job performance measurement, 
and community impact observation as you understand them? 

(2) Is the report well written; can you understand it? 
{Note. A good evaluation report should be written 
in simple straightforward, non-technical language, 
that any reader could interpret. If the report is 
laden with heavy, technical language, that only a 
highly trained person could interpret, the study will, 
in all probability, be inadequate,) 

(3) Are all the objectives for the study covered in the 
final report? 
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(4) Are the data credible to you? Do the results ^'square'' 
with your prior experience? 

{Note: If not. the study may be inadequate.) 

(5) Ar the results of the study highly unusual? 
(Note: Given the state of the art in evaluation, 
the results should uot be dramatically different 
from what you would expect.) 

(6) Does the report make highly ai-^ . . iaii statements? 
(Note: If very dogmatic statements are made through- 
out the report, he suspicious; the evaluator may be 
x-ding his own ''hobby horip^e".) 

(7) Is the report, excessively negative? 

(Nol^e: If the report is excessix^ely negative, it 
Hiay be that the ev^tiuator ±b focusing on narrow 
aspvct3 of the st7idy and not treating it as com- 
prehensively a- it should be treated; or the evalua- 
tor' s strong biafj may be showing through.) 
(8) To vhat extern t are the values of the evaluator inter- 
spersed in data interpretation? 
(Not^: To what extent does it appear that the 
evaluator made interpretations that go beyond the 
data which have been presented? Are there state- 
ments that appear to e unsupported by the informa- 
tion presented? If so, the study is probably in- 
adequate.) 
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(9) Is the tone of the report Bcientific and objective 
In its presentation? 

(10) Are the technical, data presentation aspects of 
the report presented and discussed in language 
and a context that you can understand? 

(11) Is the discussion that follows^ data presentation 
understandable? 

(a) Does it as^^ujM^ that you have an extensive 
knowledge base or does it present that knowledge 
base to you so that you cau understand the con- 
tent of the report? 

(b) Does the evaluator clearly label his inferences 
and his recomraends tions? 

(12) Does the report me the following criteria; 

(a) Does it list all the objectives for the study? 

(b) Does it specify how each objective was measured? 

(c) Does it present the results for each objective? 

(d) Does it discuss the re;- . '3 for each objec- 
tive comprehensively? 

( ) Does it summarize the findings of the study? 

(f) Are recommendations made and appropriately 
qualified? 

(13) Does the scope of the evaluation study appear to be 
consistent with :he scope of the training effort, he 
job performance ireasurement or the community impact 
evaluation? 
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Final .' nmary Statement 

After reviewing the content of an evaluation report and checking 
each of the i-ems above, tally the frequencies of "yea" and "no" responses. 
If you have a high percentage of "yes" responses in areas that are critical 
to you, then in all probability you have a high quality evaluation study. 
If your responses are predominantly "no", then you probably have an in- 
adequate evaluation study. It the study is an Important one, you may want 
to have an independent evaluator audit your findings or have a colleague 
do the sat 3. 
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6. Hew to Uee the DOES Reporting Syetem 



The vital link between e 'iluatlcn and actiofi dp^aidions 3 a decision 
information or reporting system. In the pa. nnce of communication ; ^'ory, 
the reporting system constitutes the channel between the information system 
(the sender) and the decxjion system (the user). Information on the sending 
end must be partitioned, coded, transmittedg and displayed according to the 
usev^s requirements and constraints. Thus the user should be involved at 
the outset in the design of the reporting system to insure that the informa- 
tion transmitted and displayed is maximally compatible with his sy em re- 
quirements ai;J constraints. 

Through vrorking with decision-makers and discussing their problems, 
tht need for two distinct kinds of information has been recognized: tech- 
ulcal information a^id decision information. Teahnicat information is report- 
ed in the tonn of a technical report with an accompanying technical abstract 
that records procedures, data summaries and analyses, instruments and inter- 
pretatia:iis of results. Decision information is of quite a different kind. 
I>acision infonna'^ion is the result of a translation of technical results 
into objective -c.ti'xgs ci. quality, 

Wiale ^stiona ct formating decision infonnauion have not yet been 
fully answered, )"entarive forms for reporting decision information have been 
developed as Be ■leio^ fufcr-r^^-tion Suimariee (DIS). There are five formate 
for decision iniormation. Tiif^&c are dir-played in Figures 4 through 8 • 
ctv^luatjTn iii. the training mode is (jartitloned into two formats: training 
ruaCarials evaluation and training process evaluation, lliere is one DIS 
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for job pv -:ance and one for community impact. Finally, there is a DIS 
for m l;. ,; the resu: ts of tDulti-mode evaluation into a single format to 
proviici comprehensive suimnary of the quality, costs, and recommendations 
for a specific training activity. 

In filling out the DIS, the following general sequence of steps 
should be considered: 

(1) Collect ail relevant data (information) , 

(2) Specify evaluation standards (criteria) which are 
related to the DIS ra Ing categories. 

(3) Perform consistency evaluation. 

(4) Relate drta to evaluation standards. 

(5) Enter results of evaluate on into DIS. 
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APPENDIX A 
Alternative Procedures 

(1) Alternate procedures and criteria for evaluating the quality of Training 
Materials 

Examine the training materia?.? for learner?, iustractor's manual and 

all supporting equipment. Then complete the following rating sheet. 

Criteria for the Analysis of Training Materials 

Program Gca's, Learning Experiences and Evaluation Yes No 

1, Is there ari explicitly stated general goal for the 

training package in terms of learner behavioral outcomes? 
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2. Are there specific learner behavioral objectives for 
each sub-unit of the training package? 

3. Is there a clear correspondence between the general 
goals of and the unit objectives? 

4. Do the learning e:q)eriences appear co be consistent 
with the unit objectives? 

3. Are there provisions to inform learners of the objec- 
tives which they are expected to acquire? 

6. Are learner? provided vitn an opportunity to practice 
the unit objectives they are expected to acquire? 

7. Are pi j'\rLslons made for feedback to learners regarding 
their progress ' zhe training? 

3. Are provisions ma^w ^ov feedback to learners regarding 
their achievement at the end of training? 

9. Are self-evaluation devices built in for learners to 
evaluate their progress en route? 

10. Are performcAice evaluation meai ares avail ab" 3 to deter- 
mine 1 ^mer achievement? 

11. Are performance evaluation measures congruent wi h 
unit 0^ :ictives? 

Characteristics of Instructional Materials and Activities 

1. Are the roles ^or instructors clearly specified? 
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Yes No 

?. Are the roles for learners clearly specified? 

3. I'' training provided for Instructors? 

4. Are the training materials diverr^fied with respect to 

media and instructional methods? 

3. Are the training activities diversified for learners? 

6. Are materials likely to be perceived as corresponding 

to the ievelopmental or maturity level of the learner i? 

7. Are pretests provided for learners? 

8. Can the trainees start at an advanced .^valV 

9. Coii they finish at different levels? 

Characteristics of Evaluation Materialr 

1. Are evaluation items and /or devices available? 

2. Are evaluation items related to learner performance 

objectives? 

3. Are learners given the opportunity to practice during 

training on items similar to criterion items? 

4. Are problem solving techniques used to measure 

learner behavior? 

Summarize your ratings and decide: 

1. Training materials are adequate for my needs. 

2« Training materials are inadequate for my needs. 

2. (a) Training mat::rial3 should be rejected. 

2,(b) Training materials could be modified and used with a 
reasonable amount of erfort. 

(2) An Alternate Way of Rating Training Materials 

Checklists presented in other sections of the Handbook provide the 

decir ion-maker with important characteristics of training programs/materials 

thai, should be taken into account. It might be hel-ful to present these 

rhaiacteristics in the form of Good, Better , Best for cla sification .jurposes. 
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Characteristics of training programs , A good training program will have: 
(8-24 [lOurs of inst iif. ^ ' oil) 

1. A otateraent of our,coL:^.:> in learner behavioral terms th .t: 

lend themselves to observation or measurement 
have a direct bearing to job performance 
. are consistent with the goals of training and with the training stra- 
tegies. 

2. A varlet}' of instructional materials .^n.!/cr ,liniques which activelj^ 
involve the trainee. 

A better training program (24 to 40 hou .uction) in addition to 

1 & 2 above will have: 

3. Provisions for trainees to practice (siaiuiate or role pl^y) the 
behaviors for which chey are bein. "ruined, 

4. Explicit provisions foi self-ev^rtiuatlon and feedback to trainees on 
behaviors they are expected to acquire. 

5. Built-in evaluation of training techniques which require trainees to exhibit 
skills, de .is trace knowledge ^ provide an affective response to training; 
whore possible, evaluc tion Instruments should use naturalistic and/or 
unobtrusive obse'n/ation. 

The best possible training program will h.-^ve, in addition to the above: 

6. An explicitly stated rat-fonale for the tralniro; ^;^ogram that provides 
statements of: 

documented need for the training 

as: umptions, theoir> or evidence that supports the ins'crvciLionai 
approaches used 
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evidence that '*real world" job analyses have been conducted to 
derive training objectives, 
7. A history of the development effort which includes pilot and field 

test data and course revision made on the basis of trainee performance 
behavioral data. 

Given the above characteristics, how would you rate the training program 
reviewed? 

Good 

Better 

Best 

Less than Good 

(3) Alternate Form for Evaluating Training Materials and Training in Action 

Cited from: Smith, R, G. The Design of Instructional Systems , 
Technical Report 66-18, Washington: The 
George Washington University, Human Resources 
Research Office, November 1966 • 

A Check List for Evaluating Training 

(Correct Answers Are Capitalized) 

1. Obtaining information concerning the job for which uhe 
student is being trained. 

a. Is there a procedure for obtaining information 

about the job? YES no 

b. Is the procedure applied systematically 

and consistently? YES no 

c. Does the procedure collect performance infor- 
mation for meaningful units of activity? YES no 

d. Is performance information actively sought from 

sources in the work or life performance ?iuuation? YES no 
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e. Is performance information recorded? 

£ Is performance information used systematically 
* and consistently to identify critical instruc- 



tional needs? 



Does the procedure provide complete coverage 
of all likely aspects or cccurrences of the 
desired work or life performance situation? 



h Does the procedure idc^ntify performance actions, 
condition, and standards relevant to the work 
or life situation? 

2. Identifying specific training objectives. 

a. Are decisions about what to teach made on the 
basis of reliable and valid data? 

b. Are detailed analyses made of tasks to be taught 
as a basis for identifying knowledges and skills 
required for task performance? 

c. Are all skills and knowledges required for task 
performance identified? 



e. 



3. Establishing the sequence of instruction. 

a. Is there an effective orientation of the student 
to the entire job to be learned? 



Is the relation of each new task to be learned to 
the overall job clearly stated to the student? 
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YES no 



YES no 



YES no 



YES no 



YES no 



YES no 



YES no 



d. Do training objectives et*^.- precisely the 

performance actions, cond: t^(.»B and standards? YES 



Do specific training cbjactiv2& use vague terms 

such as know, understand, appreciate, familiarize, 

general knowledge, working knowledge, qualified? yes 



YES no 



b. Are there blocks of skills and ^"^^^^^ge taught 

in isolation from their use in job tasks? 7 

c. Are new skills and knowledges taught only when 
required in order to master a new task.' 

d. Is the learning of new knowledge followed imme- 
diately by practical exercises? 



YES no 
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4. Desi lining situations for the practice ol: performance, 

a. Are practice* situations based on an analysis of 
the task to be learned? 



YES no 

b. Does the student practice the entire task? YES no 

c. Has any part of the task been omitted from practice? yes NO 

d. Do training devices simulate the task? YES no 

e» Do instructions for effective use accompany the 

training device? YES no 

f. Has the training device been evaluated in terms 

of developing student proficiency? YES no 

g. Have training devices* vs real equipment been 

subjected to cost-effectiveness analysis? YES no 

h. Has the possibility of using obsolete equipment 

to teach appropriate skills been considered? YES no 

i. Do trainees receive frequent and immediate knowledge 

of the effectiveness of their practice? YES no 

j. Do trainees receive at least one minute rest 
between practice trials? 

5* Designing situations for the practice of knowledge. 

a. Is the knowledge to be practiced clearly related 
to an actual job task? 

b. Has information representing the job cues 
provided the student, and the responses he is 

to make, been identified? YES no 

c. Has a practice session been planned for? YES no 

d. Have appropriate practice materials (workbooks, 
self-instructional programs^ flash cards, etc.) 

been designed? YES no 

e. Do trainees receive frequent and immediate 
knowledge of the effectiveness of their 

practice? YES no 



YES no 



YES no 
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f . Do trainees maintain a record of rheir progress 

during practice? YES 

6. Preparing presentationa to the student. 

a. Has the content of the presentation been tested 
on students to determine, by means of achievement 
tests, whether it communicates to the students? YES 



Is the content of the presentation meaningful 
to the student? 

c. Are there lengthy periods of presentation 
uninterrupted by practice? 

d. Are films and television integrated with 
live instruction? 

e. Are lectures, demonstrations, films, television 
or tape recordings selected on a cost- 
effectiveness basis? 



b. Have adjustments been made to the training 
schedule to account for differences in student 
aptitude? 

c. Have the interests, educational background, 
and attitudes toward formal schooling been 
determined? 



no 



no 



YES no 

yes NO 

YES no 

YES no 



no 



f . Have texts been examined to be sure tha: f,hey 

are within the reading capability of the student? yeS 

7. Maintaining student learning activity. 

a. Has the degree of spread in aptitude scores of 

the trainees been detennlned? Y£S 



YES no 



YES no 



d. Is this information used to make training 

presentations mors meaningful? YES no 

e. Do students receive rewards that are significant 

to them when they achieve course objectives? YES no 

f* Do student rewards include those that are under 

the control of the student ccapany commander? YES no 
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Haa coordination been achieved with the student 
company commander to make rewar is under his con- 
trol responsive to student performance in the 

training? YES no 

h» Are successful students punished? yes NO 

i. Are falling or borderline students rewarded? yes NO 

j« Has an analysis been made of the amount and 

reasoAS for excused absences from class? YES no 

k. Have stei><t been taken to reduce the amount of 

excused absences to a minimum? YES no 

8. Control of the quality of the training* 

a« Are the tests direct translations of the 

training objectives? YES no 

b. Is emphasis given to performance tests? YES no 

c. Are grades expressed in percentage passing? YES no 

d. Are grades based on the bell-shaped normal curve? yes NO 

e. Are grades based on percentile ranks? yes NO 

f • Are test items changed to make them easier or 
harder to conform to an "ideal" distribution 

of grades? yes NO 

g. Are results of student testing provided to the 

instructional departments? YES no 

h. Do the departments make changes in training 
procedures suggested by the results of student 

testing? YES no 

(4) Some Principles and Characteristics Related to Effective Training 

There are a great many practical outcomes from psychological research 
and training experience that provide valuable information for training 
officers in local, state cr federal gove*:nment. These outcomes ara pre- 
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sented below in the form of principles and characteristic: to the training 
officer as background information. The principles and characteristics may be 
useful in judging che quality of a) training materials, b) training-in- 
action, or c) in developing RFP's for training materials. 
Psychological Principles Useful in Training Materials Development* 

1. Learning depends upon motivation, 

2. Learning depends upon the capacity of the trainee, 

3. Learning depends upon .ue previous expei :e of the trainee. 

4. Learning depends upon perceiving relevant relationships. 

5. Learning depends upon an active search for meaning on the part 
of the trainee, 

6. Learning depends upon feedback provided to the trainee, 

7. Learning depends upon achieving satisfactory personal and social 
adjustment in the training environment, 

8. Learning founded upon the search for loeaning will be more likely 
to be repeated and applied in new settings. 

Characteristics of Effective Training Experiences* 
EFFECTIVE TRAINING EXPERIENCES: 

1. Provide the trainee with the opportunity to practice the type of 
behavior he is expected to exhibit on the job. 

2. Provide the trainee with the opportunity to deal with the knowledge 
content implied in training objectives. 

3. Are within the experience range and within the mental ability range 
of the trainee. 

* Adapted from: Tyler, Ralph W, Analysis of the Purpose, Pattern, Scope 

and Structure of Officer Education Program of Air Uni- 
versity. (AFR 190-16), Officer Education Research 
Laboratory, Maxwell Air Force Base, Alabama, 1955. 
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A. Build on the past experience of the trainee, 

5. Are satisfying to the trainee. 

6. Are perceived as relevant by the trainee in relation to his 
current Job and his career aspirations, 

7.. Demonstrate to the trainee all of the enabling objectives 
and tasks required to reach the training goal, 

8. Provide feedback to trainees en r oute regarding achievement of 
objectives so that it can be used in guiding training activity, 

9, Require good personal and social relations, 

(5) Training Systems Design and Training Materials Development: A Brief Sunmary 

Background information . To make effective decieions about training and 
training materials, the decision-maker should know something about the basic 
elements of training systems development. This basic infoirmation has been 
prepared to assist the training officer In local, state or federal goveranrent 
in judging the quality of training materials and in veloplng guidelines for 
the subcontracting of training materials,^ 

A training system is conceived of as an integrated set of resources, 
human and material, designed to achieve specified outcomes in learners. The 

^ For the reader with more than casua] interest in this particular area, the 
following readings are recommended: 

Smith, Robert G, , Jr. The Design of Instructxonal Systems . Technical Re- 
port 66-18, The George Washington University, Human Resources Research 
Office, November 1966, 

Instructional System Development . Air Force llanual 50-2, Department of the 

Air Force, Washington, D,C, 20330, December 1970, (Available from U.S. 

Government Printing Office, Washington, D,C. 20402.) 
Baker, Eva L, The Technology of Instructional Development. In Robert 

M, W. Travers (Ed.)^ Second Handbook of Research on T eaching . Chicago: 

Rand McNally, 1973, 

Briggs, Leslie J., et al. Instructiona l Medi a, Monograph No. 2, American 
Institutes for Research, Pittsburgh, 1967,^"" 
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outcomes of any training program should be a set of skilled behaviors related 
to the Job performance requirements within an occupation. The development of 
a training program must start with a thorough analysis of the tasks required 
to perform a Job, and proceed through a sequence of specific steps that ter- 
minate with verification of the training system as producing the results in- 
tended in trainees. The following model used by Air Training Command (1970) 
is illustrative of the process* 



Insert Figure 9 about here 

A brief description of the Air Force model components follows: 

1. Analyze System Requirements ; This step requires the developer to 
identify Job tasks within the larger context of the system within which the 
job is to be performed. The analysis must describe the setting within which 
Jobs are to be performed and all the interpersonal and/or man-machine rela- 
tionships related to the envlronnent of the operational system. 

2. Define Education or Training Requirements : This step follows an 
analysis of job performance requireiriGnts and identifies the duties and tasks 
requiring instrucnion. The level of difficulty of tasks, resources avail- 
able, and development time required are identified during this phase. 

3. Develop Objectives and Tests : Criterion behaviors desired as end- 
products of training are identified and eiiabling objectives (prerequisite 
knowledge, skills and attitudes) are specif iec'. Performance tests are developed 
to reflect the Job behaviors that are the object of training. 

4. Plan, Develop, and Validate Instruction ; The sequencing of instruc- 
tion, the selection of instructional materials and methods, th2 development 
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and testing of program prototypes comprise this design phase of the cycle. 

5. Conduct and Evaluate Instruction : The field testing of an Instruc- 
tional system with particular eiqphasls on data based revision or redesign of 
the Instructional sequence and materials Is a critical step. The developer 
must rework the system until It produces the Intended results with learners. 

The most critical element, then. In a training program Is the clear 
specification of job requirements, training objectives related to requirements. 
Instruction designed to produce the behavioral outcomes, and evidence that the 
behaviors have been produced. 

Smith's (1956) model of an instructional system brings one closer to the 
process of coiaducting a training program and includes some important prin- 
ciples of training. A presentation of the model and a brief discuission of 
its elements follow: (see Fig. 10) 
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Figure 10 

A Model of an Instructional System 



(Smith, 1966, p. 7) 
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From rn analysis of i:he Job performance requirements of trainees accord- 
ing to Smith's design, the developer selects specific behavioral ob jectives 
tor training program. Practice of performance refers to that aspect of trai.n- 
ing which engages the learner in the practice of the tasks and the skill 
coii5)onents of training in a Job simulation setting. Practice of knovledpe 
provides for the learner to use the symbolic processes (e.g., ke> words, 
terms, concepts, diagrams) related to Job performance and to receive feed- 
back or knowledge of results throughout training. Presentation of knowledge 
refers to the means by which knowledge is transmitted to learners. Lectures, 
reading, graphic aids, and films are amcng the most commonly used presenta- 
tion modes. Management of st udents refers to the techniques used to keep 
students participating productively in the learning experiences. Q uality 
control refers to the processes en?>loyed to gather data on the effectiveness 
of the training system in accomplishing its objectives. An equally important 
aspecc of quality control is to provide guidance regarding how the instructional 
system might be changed to increase its effectiveness. 

One other important ingredient of a training system is the specification 
of stfadards of proficiency that trainees should attain at the end of course 
instruction. These standards may also reflect a developmental process by 
which a trainee progresses through levels of job performance (i.e., appre 
tice. Journeyman, master). Proficiency levels at end of training depend upon 
the entry characteristics of the trainees and may be perceived as part of a 
longitudinal effort that will be combined with on-the-job training co ul- 
timately reach desired levels of Job performance. 
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Baclc^round Information 8uamry « The purpose of this section has been 
to provide the decision-maker with an orientation that will help him see 
the relationship of an isolated or short term training progiam, to the 
overall process of the design of m instructional system that is based on an 
analysis of job performance. These concepts are presumed to be helpful jr 
assisting the decision-maker to judge the quality of training programs and 
prepare specifications for the development of training niaterials. 
Guidelines for the .election ot Training Materials 

Background information. The guidelines that follow will provide the 
decision-maker with key ideas related to the development of training materials, 
or to the evaluation of training packages. These gu:i\delines have been derived 
from an extensive analysis of the literature on training materials development, 
and from the reported research. It needs to be mentioned that there are few 
eflq>lrical based generalizations that can be stated about the effectiveness 
of specific classes of media related to training instructional effectiveness: 
this is largely due to methodological problems in instructional research. 
For exanq>le Baker (1973) reports that there is little evidence to support the 
notion that given a media as media provides for more effective learning, 
Brlggs (1967) does point out, however, that multi-madia instruction has 
proved to be superior to 8ingle««media instruction. 

In the absence of generalizable, empirical based guidelines to support 
training materials development, one must turn to the examination of specific 
training materials that have undergone evaluation and do produce the desired 
results in lecmers. Examination of these materials provides a basis for 
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development and also provide guidelines criteria that could be applied to 
the analysis of training materials • 

In summarizing the available literature on training and in drawing 
extensively on the training research experience of HumRRO, Smith (1966) 
stater^ some guidelines related to thia instructional system design: 
"The critical aspects of practice of performance are: 

a. to simulate the job taak, using a detailed description 
as a guide 

b. to provide for knowledge of results 

c. to arrange a suitable practice schedule 

d. to maximize the transfer of training. 

The critical aspects of practice of knowledge are: 

a. tc determine, through analysis, the relation between 
cues and responses required by the knowledge 

b. to develop, through practice with knowledge of re- 
sults, a high level of achievement 

c. to devise ways of making material meaningful to 
trainees • 

Presentation of knowledge can be done successfully by any of several 
methods provided: 

a* the presentation communicates to the student 

b. the material presented is meaningful 

c. the special characteristics of media are taken into account. 
Instructional devlcea and media should be selected in terms of cost and 

effectiveness. " 
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The Air Force Manual on InBtructlonal System Development (AFM 50-2) 
provides an excellent overview of the principles and practices that one 
ought to take into account when planning, developing and validating instruc- 
tion. In presenting options to be pursued in the selection of media, the 
manual presents a very useful adaptation of a media selection matrix developed 
by Allen (AFM 50-2, pp. 5-14) (see Table 1). 

The ratings of high, medium and low in the table above referred to 
indicate the degree of suitability of the instructional media when compared 
to the objectives of learning. 

Given this brief overview of characteristics of training systems, the 
following criteria will prove to be helpful in analyzing the potential 
effectiveness of training materials and/or training programs. 



Ill 



106 



TABLE 1 

Instructional Media Stimulus Relationships to Learning Objectives 



OF 

ITRUCTIONAL ' 
^lA 


LEARNING OBJECTIVES: 


Learning 
rac uuax 
Information 


Learning 

V-f ana 1 
V XoUa J. 

Identifi- 
cations 


Learning 
IT r xncxpxeSy 
Concepts, 
and Rules 


Learning 
Procedures 


Performing 
DKXxxea 
Perceptual- 
Motor Acts 


Developing 
Desirable 
Attitudes, 
Opinions, & 
Motivations 


H Pictures 


Medium 


HIGH 


Medium 


Medium 


low 


low 


ion Pictures 


Medium 


HIGH 


HIGH 


HIGH 


Medium 


Medium 


evision 


Medium 


Medium 


HIGH 


Medium 


low 


Medium 


iniiDft Aids 


low 


HIGH 


Medium 


Medium 


low 


low 


io Recordings 


Medium 


low 


low 


Medium 


low 


Medium 


iner 
nulator) 


Mediiim 


HIGH 


HIGH 


HIGH 


HIGH 


Medium 


granmed 
traction 


Medium 


Medium 


Medium 


HIGH 


low 


Medium 


onstration 


low 


Medium 


low 


Medium 


Medium 


Medium 


nted Textbooks 


Medium 


law 


Medium 


Medium 


low 


Medium 


L Presentation 


Medium 


low 


Medium 


Medium 


low 


Medium 



i&«-^Thi8 table is adapted from Dr. Allen's paper, ^'Research in Instructional Media and Art 

Education," which was originally publisned in August 1966 in B'inal Report of the Uses 

of Newer Media in Art Education Proj ect by the National Art Education Association, NEA, 
Washington, D« Cc 
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(6) Alternate Guidelines for Subcontracting Development of Training Materials 
Given the brief cvervlev to Instructional or training system development, 
what should you do as a decision-maker to Insure the development of quality 
training materials? The following list suggests some things you could do and 
what you should expect from your subcontractor to achieve good tralnliig 
materials* 



Things you should do: 



1. Provide developer with a 1, 
statement of the general 
goals you want developed 
in trainees • 

2« Provide developer with a 2« 
sample of a specific ob- 
jective of training. 



3. Provide developer with: 3. 

a) description of training 
population 

1) age 

2) educational background 

3) experience 

b) statement of length of 
training in hrs/days 

4. Provide developer with a 4. 
list of characteristics of 

the training program you 
desire: (e.g. Training 
sessions must provide for: 
problem solving activities; 
small group discussion; 
simulation of Job perfor- 
mance; paper and pencil 
work exercises). 



Things you should expect of your 
development subcontractor 

Provide a job analysis of the 
trainee's occupation with 
specific reference to your 
goals. 

Provide list of training 
objectives that specify condi- 
tions , performance actions , 
and standards to be attained 
at end of training. 

Provide you with a tentative 
schedule of training activities 
to fit your available training 
time. 



Provide you with a tentative out- 
line of the type of learning 
activities to be Included in 
training for your approval. 
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Provide developer with a 
statenent for delivery of 
training package. 



Sign contract and monitor 
work* 



Provide you with a time table 
with milestones (completion 
of key events) specified and 
key review points for the 
development work* 

Provide you with periodic re- 
ports and review points to 
assist in monitoring the 
quality of work. 
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APPENDIX B 

Specimens of Measmrement Instruments for Pilot lestlng 
(Workshop Questionnaire) 
Prc-sesslon Survey 
It Is anticipated that there will be a continuing series of workshops 
given to aid local community leaders In administering the Revenue Sharing 
program. To make these workshops effective, we are asking you to respond 
to the following quest Icms which will help us to evaluate our efforts. 
Please do not put your name on this paper. 

Instructions 

Please respond to each statement or question by circling the letter 
(a. 9 b,9 c, 9 dm 9 or e,) that best describes your position, attitude, or 
feeling, 

1, Please check the type of goveiiimental unit you represent: 

T own, City, County, Council of Governments, 

Social Service District, or ^Other , 

(write in) 

2, Please check the population of the unit of government you represent: 
less than 5,000, 5,000 to 10,000, 10,000 to 50,000, 

* 50,000 to 100,000, over 100,000, 

3, Please check only one item in this question. 

With respect to Revenue Sharing my job is one in which I am concerned 
primarily with: 

m aking decisions, m aking recommendations, ^implementing guide- 
lines, k eeping records or accounts, e nforcing regulations. 
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4. If my co-workers were to aak me a question about whether or not Revenue 
Sharing funds could be spent for a specific item, I would feel 

in answering their question. 

a) very b) some c) neither d) uneasy e) very 

confident confidence confident uneasy 

nor 
uneasy 

5. The revenue Sharing materials I received prior to this workshop helped 

nie In understanding how Revenue Sharteg can be applied to my 

community. 

a) a great deal b) somewhat c) very lit: tie d) not at all 

6. My understanding of the Revenue Sharing '^raldellvfes as they apply to 
my situation is . 

highly b) somewhat c) aeltzher d) somewhat e) highly 

adequate adequate adequate inadequate inadequate 

nor 

Inadequate 

7. When I think about deciding or recommending how to spend Revenue 
Sharing funds, I feel . 

a) highly b) uneasy c) neither d) confident e) very 

uneasy uneasy confident 

nor 

confident 

8. In comparison to other federal programs. Revenue Sharing seems to have 
"red tape", based on my experience. 

a) much b) a little c) about the d) a little e) much 

more more same amount less less 

9. My general feelings toward Revenue Sharing are . 



a) very b) somewhat c) neither d) somewhat e) very 

negative negative negative positive positive 

nor 

positive 
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10. I feel that workshops of this kind are in assisting local 

government officials , such as myself » to understand how Revenue 
Sharing applies to community needs. 

a) very b) somewhat c) neither d) somewhat e) very 

effective effective effective ineffective ineffective 

nor 

ineffective 



11. I feel that my knowledge of Revenue Sharing is . 

a) very b) somewhat c) neither d) somewhat e) very 

adequate adequate adequate inadequate inadequate 

nor 

inadequate 

12. What is the one thing about Revenue Sharing which concerns you most? 
Please write a short response. 
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(Workshop Questionnaire) 
Post-session Survey 
We are again asldng you to respond to a few questions to see if the 
workshop has met your needs. The information you provide us by your 
responses will be used to in5>rove future pi^sentations. Please do not 
put your name on this paper. 

Instructions 

Please respond to each statement or question by circling the letter 
(a. 9 b.y c.y d.» or e.) that best describes your position, attitude or 
feeling. 



1. In general 9 I feel that the Revenue Sharing workshop was . 

a) highly b) somewhat c) neither d) somewhat e) highly 

effective effective effective ineffective ineffective 

nor 

ineffective 



2. In coiq>ari&on with categorical funding programs I believe that Revenue 
Sharing programs will have flexibility. 

a) much b) a little c) about the d) a little e) much 

more more same amount less less 



3. In terms of the questions that I had, I feel that this workshop gave 
answers to my questions. 

a) completely b) somewhat c) neither d) somewhat e) completely 

Inadequate inadequate inadequate adequate adequate 

nor 

adequate 



118 

o 

ERIC 



113 



I feel that the length of this worlcshop was 



a) highly b) somewhat c) neither d) somewhat e) highly 

adequate adequate adequate inadequate inadequate 

ncr 

inadequate 

If I were asked to expend or recommend for expenditure of Revenue Sharing 
funds for some project right now, I would feel • 

a) very b) somewhat c) neither d) somewhat e) very 

uncomfor- uncomfortable uncomfortable comfortable comfortable 
table nor 

comfortable 



Now that the workshop is over, I feel that my knowledge of Revenue Sharing 
is . 

a) very b) somewhat c) neither d) somewhat e) very 

inadequate inadequate inadequate adequate adequate 

nor 

adequate 

I feel this workshop provided opportunity to ask the questions 

I wanted to ask. 

a) more b) somewhat c) neither d) somewhat e) completely 

than adequate adequate inadequate inadequate 

adequate nor 

inadequate 



Now that the workshop is over, my general feelings toward Revenue Sharing 
are • 

a) more b) somewhat c) neither d) somewhat e) more 

negative more negative negative more positive 

nor positive 
positive 

Right now, I feel that I have knowledge to explain to my co-workers 

how Revenue Sharing regulations will be applied in our community. 

a) coiiq>letely b) somewhat c) neither d) somewhat e) completely 

sufficient sufficient sufficient insufficient insufficient 

nor 

insufficient 
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10. The dlsctisslons about Revenue Sharing helped me in understanding 

how Revenue Sharing can be applied to my coaanunity, 

a) a great deal b) somewhat c) very little d) not at all 

11. Now that the workshop is over, what concerns you most about Revenue 
Sharing? 

Please write a short response. 
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(Training Evaluation Pilot Teet) 
Background Inforxoatlon 
Principles of Supei^sion 
(Do not put your name on these sheets) 



Age: 

Sex: 

How many years have you been a supervisor: 
How many people work for you: 



What kind of work do you supervise: 



How many supervision courses have you had before this course: 

Principles of Supervision 
A supervisor should (check the most inqportant sentences): 

spend most of hf.s time checking work done by others e 

make sure employees understand work rules and policies. 

correct an employee in front of others if needed. 

try to get other employees to do his paper work. 

give up the quality of work to get a greater amount of work done, 

help enq)loyees to do their work when needed. 

pass most work problems "upstairs" for action. 
let someone else worry about how well his employees are trained. 

pick only the best enqployees for overtime work. 
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pay no attention to gripes of employees • 

explain city policy to enf>loyee8* 

cause employees to gripe about city policy • 

Job Functions 

In the list below, check those jobs a supervisor should do: 

make sure work Is done on time. 

check finished work of employees often. 

train clerk to use new office machines. 

encourage secretaries to try to make fewer mistakes In typing letters. 

arrange for machine repair ahead of time to lower the amount of 
worker down time. 

arrange work to make sure the paper work gets done on time. 

type finished copy of safety report to give to management. 

explain the job of. the supervisor in passing on gripes to management. 

keep quiet about who arranges work and work plans. 

pay no attention to employee gripes about noisy working places. 

decide the steps to be followed in recording how money is spent. 

decide that secretaries In his department ccn leave work 30 minutes 
early to miss rush hour traffic. 

decide to buy a new kind of electric machine without checking with 
management. 

unload trucks to help out workers. 

decide to cut down trees along a street without asking management. 
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start new coffee break policy even thou^ the management does not 
agree with It. 

carry out regular check to find possible safety problems. 

listen to and put to work secretaries' ideas to malce work better. 

answer questions that the public asks about cnanges in tax rates in a 
nice and careful way. 

set work standards for his department. 

check with new employees to make sure they have enough information 
to fill out personal leave requests. 

gather facts for management about possible cost-iiavlngs by using 
new machines. 



Situation Number One 
Joe is a machine operator lu a shop. His work is simple, and he is 
paid by the hour. He has always done his work well and has never caused any 
trouble. For the past two weeks you notice that his work output has been 
getting worse each day, while the work output of the men around him did not 
change. As his supervisor, you 

(put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. Tell Joe to find another job. 

2. Talk to Joe's co-workers to determine why his work output is down. 

3. Warn Joe that if his work output doesn't improve you will fire him. 

4. Check with Joe's family to find out if everything is okay at home. 

5. Talk with Joe about his work output. 

6. Talk with management about your problem with Joe's work output. 
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7. Put Joe on another Job. 

8« Invite Joe out for a beer to talk with him about how things are going 
In the shop, 

9* Encourage Joe to tell you of his problems as soon as they arise c 

Situation Number Two 
As fo reman y you are faced with a group of 14 employees who Indicate 
they have some grievances y however , you refuse to talk to them as a group • 
Instead you suggest that they appoint a spokesman. Before you leave, you 
threaten disciplinary action unless they are back at work In two minutes. 
When you return and find them still gathered , you 

(put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. Tell all the men to clock out and then you will discuss their grievances 
with them. 

2. Tell all the men to clock out and go home; they are suspended for 
two days. 

3« Talk to the men about their grievances. 

4. Set up a time to meet with the men and then tell them to get back to work. 

5. Tell the men to go back to work; that you will talk with each man 
separately. 

6. Admit that the disciplinary threat was a mistake and promise to listen to 
their grievances first thing tomorrow morning. 

?• Pick out the leader of the group and chew him out right there. 

8. Pick out the leader of the group , fire him, and then threaten to fire 
anyone who is not back to work within two minutes. 
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9. Tell the men to forget their grievances because management won't listen. 

10. Ask the men to choose a grievance committee of three to tell you their 
problem and ask the other men tc go back to work. 

Situation Number Three 
An enq)loyee, afraid to follow your instructions about a new method of 
loading a fork lift, talks to another super'/isor who informs him that he 
does not have to do the Job if he feels it is unsafe. As supervisor you 

(Put a plus (+) by the two things you would be sure to do and put a 
minus by the two things you would be sure not to do.) 

1. Go to the shop and chew out the employee for not following your orders. 

2. Talk to the employee to try to find out why he thinks the new method is 
unsafe. 

3. Get someone else to load the fork lift. 

4. Chew out the other supervisor for undercutting your Job. 

5. Get an outside expert to say how safe the new method is. 

6. Talk to your boss about the problem to find out what he would do. 

7. Try the new method yourself to find out how safe it is. 

8. Talk to both, the employee and the other supervisor together to find out 
exactly what the problem is. 

9. Threaten the employee with disciplinary action if he doesn't do the Job. 

10. Encourage the employee to talk directly with you rather than take his 
problem to some other supervisor. 

Situation Number Four 
Tour boas calls for you to report to his office. As you walk in, 
you can see that he is very busy. He hands you an unfinished work order, and 
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"Take as many man as you need off what they are doing and get this job 
out". The phone rings, he answers the phone and says, "Yes sir", gets up 
and walks out leaving you standing there. This is about the fifth time in 
the past few weeks that he has failed to give you complete instructions. 
You decide that you should 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. Get to work, obeying his orders to the letter. 

2. Sit down and wait until he comes back so that you can get further in- 
structions. 

3. Call in one of your workers and give him the same instructions. 

4. Go back to your office and hold the work order until you get the 
whole story. 

5. Write a memo to your boss that tells him how difficult it is to do a 
good job without complete Instructions. 

6. Start the work and then try to get more complete instructions. 

7. Call in a worker and have him wslt to get complete instructions from 
your boss. 

8. Send a memo to your boss's superior telling him about the poor in- 
structions you get. 

9. Call in a worker and complain to him about how bad your boss is. 

10. Follow your boss and demand that he give you all the details right 
then and there. 



Situation Number 
As supervisor, you have just spent 
ties of street repair work for your city, 
office at which time you are told of many 
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a lot of time planning the priori- 
You are called into your boss's 
complaints about unrepaired streets 
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called In by citizens. You notice that these complaints came from places 
in the city that you gave the lowest priority for work. You decide you 
should 

(Put a plus (+) by the two things you would be sure to do and put a 
ndnus (-) by the two things you would be sure not to do.) 

1. Forget your plana and begin work on the streets complained about. 

2. Show your boss your earlier plans and make a strong case for following 
them. 

3. Ignore the coii^>lalnts and follow your earlier plans* 

4. Ask your boss to help you rep Ian the street repair priorities. 

5. Ask your boss to give you guidelines to set priorities for street repair 

6. Do nothing until you receive direct orders about what streets should 
have priority for repairs • 

7. Gripe at your boss for not giving you the complaints sooner so that 
you could use them in your planning, 

8. Ask your boss to decide what streets should be repaired. 

Situation Number Six 
Your boss calls you into his office. He tells you that it looks like 
your men have not been performing their work up to standard because he has 
received many complaints from citizens. As a supervisor, you have carried 
out weekly inspections of the work and found the quality to be good. At 
this point you 

(Put a plus (-f) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do. 

1, Tell your boss that he does not have the facte o 
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2. Try to find out what the problem is. 

3. Ask your boss to cone with you and Inspect the work. 

4. Feel bad because your boss says that you haven't been doing a good job. 

5. Decide that you will have to spend much more time Inspecting work. 

6. Decide that you will have to train the men to do better work. 

7. Chew out your men for their poor work. 

8. Decide to set higher standards for quality of work. 

9. Ask to speak personally to citizens when they call in complaints. 

10. Question employees about who is not doing good work. 
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( PERFORMANCE RATING OF HOUSING MAINTENANCE PERSONNEL ) 



The purpoae o^' this rating check-list is Lo furnish ^lil/i la J>valuatc the 
effects of training and assess training needs toward the I'pd of .isaisting tliu 
San Antonio Housing Authority meet Its training objectives. This check-list 
does not have as Its purpose the evaluation of Individuals for placement or 
advancement i because of this, all Information contained on these forms is to 
be held In the str'.ctest confidence. 

Name of Housing Project Name of Maintenance Aide or Mechanic 



Name of Supervisor Filling Being RateH 

Out This Form 

Circle one: Aide A Aide B Mechanic 

Date 



Instructions ; On the following forms you will find a list of tasks that 
might generally be performed by a maintenance aide or mechanic In the housing 
authority. At the right of each task statement you are asked to check whether 
you have observed the performance of each task within a 30 day period or whe- 
ther you have ever observed the performance by the man you are rating. Finally, 
you are asked to make two ratings of the man's performance of each task on the 
basis of the amount of supervisory help he requires and his proficiency in 
performing each task. Amount of Supervisory Help falls into three categories: 
Little, Moderate, and Much; Proficiency In Performing the Task also has three 
categories; Highly Skilled, Fairly Skilled, and Needs Skill Development. Each 
category Is described below to aid the supervisor in making his ratings. 
Please read the descriptions for the rating categories carefully before pro- 
ceeding with the task ratings* The descriptions should be referred back to in 
filling out the rating form. 



1. Amount of Supervisory Help ; 

Little : Once this man is given the task, he requires little 
or no further instruction. 

Moderate: Once this man is given the task, he needs some or 
occasional instructions to carry out steps in per- 
forming task. 

Much: This man needs almost continual, step-by-step Instructions 
to do task. 



2, Proficiency in Performing the Task ; 

Highly Skilled : Once this man performs task, supervisor is 
confident that the task need not be redone. 

Fairly Skilled ; On checking the task performed, supervisor only 
occasionally finds that the task needs to be 
redone. 

Needs Skill Development ; On checking the task performed, super- 
visor frequently finds that the task 
needs to be redone. 
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